In my opinion you could mount a very compelling argument on this if you can actually prove and get admitted into evidence what you say. So, the answer is yes it is possible.
The remedy is usually not losing rights, but rather to modify visitation in a way that protects the children. If his visitations are supervised that already may be the case. If the restrictions need rewriting, see a lawyer about modification. The facts you give, if proven, might give grounds to alter visitation.
Sign up to receive a 5-part series of useful information and advice about child custody law.