Just because somebody signed the birth certificate doesn't mean they are granted custody. There needs to be a custody agreement in place, either agreed upon by the parties or Court Ordered. The Courts do not favor the Mother of the Father, at least they are not supposed to. Looks like your case has complicated facts so I suggest spending the money for a consultation to get a game plan together.
Sign up to receive a 3-part series of useful information and advice about child custody law.