"spousal privilege" is typically referred to in a criminal setting where one spouse can not be made to testify as against the other. Even if arguendo applied in a civil setting, it would not help you because your comments to the other woman's husband calling her an adulterer was effectively a published statement which, if not true, is defamatory, and is not "privileged" since you are not a "spouse" to her husband and therefore, does not make you immune from a defamation action. Note that "truth" is a defense, so hopefully you can, if forced, prove that an affair was in fact had between them - something more than e-mails (unless one or both admitted to the act therein) would be necessary. On the other hand, she will have to think twice about following through with her threat, as such cases tend to open secret closets and are expensive to prosecute.
A roundup of the best tips and legal advice.