The answer depends on whether there are any orders here in Virginia for custody and visitation. You can try to take him to court to stop this, but unless it hurts the children and they have separate rooms, you will have a hard time convincing a judge to order that no one of the opposite sex be allowed to stay in the house while they are there.
Consult a local family law attorney before taking any action.
[This communication is intended as general information and not specific legal advice, and this communication does not create an attorney-client relationship.]
Sign up to receive a 5-part series of useful information and advice about child custody law.