You should meet with an attorney regarding your options. It is not really a domestic abuse situation but I can see your concerns and I agree that your concerns are reasonable. An attorney can help you sort out your legal options. How old is your son?
Best of luck to you.
This response is intended to be a general statement of law, should not be relied upon as legal advice, does not create an attorney/client relationship and does not create a right to continuing email exchanges.