You do not designate the forum in which your ex is saying he will "file charges." I can tell you that in custody cases, all kinds of accusations are thrown about. people threaten a lot, but unless it's in a police report or a pleading, ignore it. I do not know your ex's motive: It could be your ex honestly believes he is protecting his children, or it could be that he is trying to leverage some advantage against you. Your best bet is to be honest - is there was an issue, own it; if there was not an issue, (and he files something) look into a psych evaluation or a custody evaluation. Let the experts determine the merit.
If he is threatening criminal charges, remember that your ex doesn't get to "file" any charge. He can make a complaint with law enforcement, but it is up to them to investigate and then refer the matter to the District Attorney if they believe the allegation has merit. It is then up to the DA to decide if charges will be filed. If he is making the threat in the custody proceedings, it is up to him to prove the allegation. If he fails to prove it and really has no evidence, the court will not be amused.
Sign up to receive a 3-part series of useful information and advice about child custody law.