There are a few things you can do to help your situation. First, document every instance of this kind of misconduct. Just jot down the date, time, and what happened on a piece of paper, and keep that sheet safe. This will help you remember exactly what happened, and you'll sound much more credible if you have specific dates and times to tell a judge. Second, you may be able to file for a protective order. You can contact either your local police department, or the clerk of the Juvenile and Domestic Relations court in your jurisdiction for more information about this process. In the end, if your ex has been granted visitation with your children, you may not be able to keep him away from them, but if they are upset by his actions, a protective order will at least stop the questionable behavior. Third, you might ask for a Guardian ad Litem to be appointed to your custody case. The court can appoint this person in cases where there are needs that are not being addressed by the lawyers for the parents. The GAL acts as a lawyer for your children, and can request to be around your ex when he's with the kids. Fourth, you may be able to file for supervised visitation. Supervised visitation means that a trained therapist is with your ex and the kids at all times, and would be able to report to the court if your ex continues to say these things.
Remember, these are just general suggestions, and may not necessarily apply to the specific facts of your case. It may be worth getting in touch with the legal aid organization in your area, or an attorney who specializes in family law in your area.