If the ex does raise the issue, be prepared to cooperate with a stepwise reintroduction. A little professional coaching may be of help. Most professionals would suggest it is in the best interests of the child to foster contact with his mother rather than "protect him from visitation" but the reintroduction could helpfully involve some supervision, shorter and more frequent visits, etc.
Best wishes for an outcome you can accept, and please remember to designate a best answer.
This answer is offered as a public service for general information only and may not be relied upon as legal advice.
Counsel gives you excellent advice. I would add that you should also check your custody agreement to determine how many times mother has not met her parenting responsibilities. You can generally use that as a tool in requesting supervised visits. Good Luck.
Legal disclaimer: The response given is not intended to create, nor does it create an ongoing duty to respond to questions. The response does not form an attorney-client relationship, nor is it intended to be anything other than the educated opinion of the author. It should not be relied upon as legal advice. The response given is based upon the limited facts provided by the person asking the question. To the extent additional or different facts exist, the response might possibly change. Attorney is licensed to practice law only in the State of Massachusetts. Responses are based solely on Massachusetts law unless stated otherwise.