The answer to the hypothetical question is yes. However, obviously there are many more facts than those in the posting that would determine the outcome. This answer is provided for informational purposes only. Actual legal advice can only be provided in an office consultation by an attorney licensed in your jurisdiction, with experience in the area of law in which your concern lies.
Your brother would have to report that you are unfit to the Department of Child Services in your area. They must conduct an interview with you regardless of whether or not there is suspected abuse. They will come out to your home to interview you and the child. If they find nothing, the case is closed. If they suspect neglect, abuse, etc., they will then make a report and they can file to have your child put in foster care or use another relative as a resource. Do not let your brother intimidate you. If there is nothing going on in your home, his threats are meaningless. He has no standing to sue for custody and unless the Department of Children and Youth find an issue, it will go no further.
Sign up to receive a 5-part series of useful information and advice about child custody law.