Yes and no. It is possible for one parent to request that there be restrictions about who the other parent exposes their children too. However, the restrictions must either be agreed to voluntarily by the parents or ordered by the judge during a custody battle. For the judge to order such a restriction, it must be shown that it is needed to protect the child's best interest. (Thus, a restriction is not likely to be granted just because the father does not like the person.) Finally, such restrictions are usually either reciprocal (e.g. neither parent will expose the child to anyone with a record of using or distributing illegal substances) or specific to a person with a relevant history (e.g. Parent A will not leave child in the unsupervised presence and custody of Person P who has a prior conviction for being a child predator).
If there is not a current custody battle in place, then the short answer to your question is "No". Neither parent has any right to dictate what takes place in the other parent's home or during the other parent's custodial parent (aside from specific provisions within custody orders.)
The above answer is a general explanation of legal rights and procedures. It does not constitute legal advice. Nor does it establish an attorney-client relationship between the individual posting the question and the attorney providing the answer.
The quick answer is no. He may attempt to exert control or make threats of legal action but if what you say is true, there is no grounds. He has no legal say.
This response does not create an attorney client relationship nor should the advice be relied upon because it is not specific to your legal situation. Before you depend on legal advice, you should retain competent counsel.