In my state (Washington), this would fall under the intentional tort of intentional infliction of emotional distress. Check with a local attorney to see whether your state has a similar intentional tort.
Some States recognize a "pastor-penitent" privilege which prevents the courts from requiring a pastor to testify regarding what a person confesses in a confidential setting.
As far as I know, there is no special privilege which allows pastors to commit defamation of character.
Perhaps you should call an attorney to discuss the details of your particular circumstances.
[In accordance with the Avvo community guidelines, this communication does not constitute "legal advice", nor does it form an attorney-client relationship.]
Sign up to receive a 3-part series of useful information and advice about personal injury law.