This is a general answer and you should always speak to an attorney in your area. The Court always looks at the best interest of the child. There is a basic presumption that both parents are fit to have custody. Then you look at each party individually. In your case if the father never comes around the child and you are not preventing his access, then the Court will take that into consideration. If a father wants to see the child and he is deemed a fit parent the Court gnerally will allow visitation. Whether he is married or not generally is not a factor, unless the child would be at risk with his spouse.
Sign up to receive a 3-part series of useful information and advice about child custody law.