Truth is an absolute defense. Then again it will be your word against his regarding what was said.
DISCLAIMER: David J. McCormick is licensed to practice law in the State of Wisconsin and this answer is being provided for informational purposes only because the laws of your jurisdiction may differ. This answer based on general legal principles and is not intended for the purpose of providing specific legal advice or opinions. Under no circumstances does this answer constitute the establishment of an attorney-client relationship.
If you are send the letter back to the person who said these things, it would not be defamation, which only applies to statements made to third parties. If you are cc'ing other people in on the conversation, I'd leave out his statements: they won't help you win. Saying you are "disappointed" or that he was "unprofessional" are statements of opinion and are likely not defamatory. The key in these situations is to try to take emotion out of it. If you do that, you probably won't get yourself in trouble.