Another issue to consider is that fraud is a two way street. In the event that he claims that you committed fraud by getting married (my assumption is that he is claiming that you are defrauding the government by marrying just to obtain citizenship) what you have to understand is that he is complicit in that fraud as well (i.e. he was aware of the crime and did nothing to report it). This leads him potentially on the hook for criminal prosecution as well. My guess is that he is aware of that and will not take any action against you, especially because the action would be potentially double-trouble for him by reporting a fraud that did not exist, i.e. making a false police report (also a crime).
If you are a US citizen, and your marriage was not fraudulent, then you will not have your citizenship revoked if your husband wrongfully accuses you of fraud. If he does accuse you of fraud, it will be his word against yours. If he has no evidence other than his word, then it is very unlikely that you will be found to have committed fraud. And if he supposedly knew that the marriage was fraudulent when you got married, then he can go to jail too for participating in the fraud.