My doctor never took me seriously, so I wrote an angry letter to him (e.g. I had acute Congestive Heart Failure for four years and he had never tested me for anything-even an EKG. I personally did not think the letter was harassing. He just would not talk to me about it personally, so I wrote it down. I marked the letter Personal. Later on, I found that he had instructed all the staff in his office-it was a big office (and a call center) to harass me. Does this come under the privacy act? He is now retired. Was it unethical?
I am going to assume for purposes of answering your question that by privacy act you mean HIPPA laws which generally protect your medical information from being disseminated without your permission, and I am not sure if your letter would fall under that and even if it did, he only shared it with his office. If you have evidence that your doctor instructed his office to harass you or committed malpractice in failing to treat you as other like professionals in his field would have, then you may want to consult an attorney about a possible claim against the doctor and his office. If you feel that what your doctor did was unethical you may want to contact the state or federal regulatory board that oversees doctors, such as the American Medical Association.
HIPAA has a "minimum necessary standard" which limits how much protected health information is used, disclosed, and requested for certain purposes. Generally within an office (at least based on my understanding) the staff may only discuss a patient's health for "treatment, payment, or health care operations" 45 CFR 164.502. From what you say, it definitely doesn't sound as thought it was for treatment or payment purposes. It may fall under the "health care operations" heading but probably not.
Depending on the type of harassment you may want to contact a lawyer for a free consultation and/or for medical malpractice if you suffered damages as a result of his failure to diagnose.