It matters. If the person asserts his or her Fifth Amendment right not to testify, based on a good faith belief that he or she may face criminal liability, the court should respect that. The most common context is where both parties in a domestic violence situation were hitting each other and the "victim," the girlfriend or wife, then comes to court and is concerned about a battery charge against herself.
The marital privilege also grants a blanket privilege to a witness to refuse to testify against his or her spouse. (Evidence Code sections 970 and 971). This deals more with a spouse trying to keep intermarriage communications, not conduct, confidential.
Talk to an attorney about this and give him the facts. More facts will give you more clarity on your options and rights.
The fine for contempt is usually $500, but it can be more.