Highly unlikely. The dad is unemployed and homeless, and has only seen your daughter 5 or 6 times. You should file a petition to establish a residential schedule; ask for custody for you, and brief supervised visitations for him.
DISCLAIMER: This answer is provided for general educational purposes only. By using or participating in this site you agree and understand that there is no attorney client relationship between you and the attorney responding, and no attorney-client confidentiality. The law changes frequently, and varies from jurisdiction to jurisdiction. The information provided in this answer is general in nature and may not apply to the factual circumstances described in your question. The applicable law and the appropriate answer may be different in the State or States where the relevant facts occurred. For a definitive answer you should seek legal advice from an attorney who (1) is licensed to practice in the state which has jurisdiction; (2) has experience in the area of law you are asking about, and (3) has been retained as your attorney for representation or consultation. Your question and the attorney’s answer may be used for promotional or educational purposes.
When this sort of question pops up, seems as though it is occasioned by the other parent saying, "I have rights too! And if you ask for child support, then I'm going to get custody!" This is something any judge has seen before and it is not a persuasive argument.
That said, any parent does have a constitutionally protected liberty interest in parenting their child unless or until a court says their rights are limited or restricted. The current trend for babies/toddlers is to have short periods of residential time with both parents, alternating. This is because doctors/psychologists say that the child needs frequent contact with both parents in order to bond with them, and it is in the best interests of the child for both parents to have time with the child. You can argue that "he's homeless and jobless", but that doesn't mean he's unfit per se. Perhaps he has a relative where visits could happen. Joblessness is not a disqualifier for residential time - LOTS of people are out of work right now. Doesn't make them unfit parents.
If you are seriously worried that he's going to conceal her away from you then you will have to be able to articulate the underlying reasons why. Has he done this in the past? Does he have another child from another relationship and that child's mother cannot find her child? You have to give the court more than a hunch.
Best thing would be to get a family law attorney to help you draw up a parenting plan and a support order. If you have a factual meaningful basis for why his time should be restricted or supervised, you can ask the Court to appoint a Guardian ad Litem for your daughter. The GAL will investigate and advise the court of the facts so the judge can best decide what to do.
Family law probably looks easy and self-explanatory, but it isn't. The situation you are describing is a really good reason to hire counsel, prepare your case, and get orders consistent with your child's best interests. And never, ever take legal advice from your ex.
Hope this helps. Elizabeth Powell
Sign up to receive a 5-part series of useful information and advice about child custody law.