I have seen parents in your situation lose custody when the judge became convinced that the one parent was preventing the other parent from having child access for no good reason. Whether you realize it or not, you are in a risky and precarious position. You must take action to make sure the court does not come to the conclusion you are cutting off the mother from her son. I recommend you hire an attorney to help you with the necessary legal battle to make sure the mother's attorney does not persuade the judge to go against you.
This is NOT legal advice, is GENERAL INFORMATION ONLY, and does NOT establish an Attorney/Client Relationship with you. Therefore my answer cannot address your specific legal situation and you should not rely upon my answer in your legal matter. I am an attorney licensed in Maryland and California.