As there is a co-parenting agreement he has the right to request custody in court. You need the aid of an attorney. The method of conception in this case is immaterial.
I am an attorney with over a decade of experience in Matrimonial and Family Law with offices in Brooklyn and Manhattan. I have experience in all five boroughs as well is Nassau and Suffolk County. The opinion expressed in this ad based upon the limited information provided and do not indicate an attorney-client relationship
There's not much you can do about his threats. I would, on the other hand, keep a careful paper-trail with him and document via letters/e-mails to him all important conversations & exchanges. Schedule a consult with a Child Custody lawyer in your area.
* If you found my answer to be "HELPFUL," or the "BEST ANSWER," please feel free to mark it accordingly.