If you have an attorney notify him/her of the actions of your spouse and her paramour. Your attorney can seek a restraining order and request an injunction effectively forbidden the behavior. Also, a protective order may be sought from the Court.
If you are pro se you can go to the District Attorney of your County and ask for a protective order or contact your local police department and report the actions to see if they will assist you in some manner.
Be aggressive in seeking protection.
Defamation is a tort action that you can file and have joined with your divorce against your spouse which will muddy the water so much for your ex giving you a greater chance to receive a more favorable outcome at trial.
YOu may consider a protective order. YOu would have to show family violence has been committed and is likely to be committed in the future. Instead of the protective order, you may also consider a temporary restraining order.
It doesn't sound like your wife or the boyfriend have threatened physical harm which is necessary to obtain a Protective Order. You didn't say whether you have an attorney. If you do, let your attorney know about this behavior. If you have children this type of accusation can effect your ability to have custody of your children or restrict your visitation with your children. This is a serious matter.
I recommend that you consult with an attorney regarding this matter.