The short answer: you can ask the office to stop contacting you at work. These requests should be done in writing. You can file a complaint with the board that supervises the particular doctor (M.D. or D.O.). You could also file a complaint with the Department of Health and Human Services. However, none of these will result in money damages payable to you. They could potentially result in a brief investigation.
As with most legal questions, you should ask yourself, is this worth my time and energy?
There is no cause of action imposing liability for failing to immediately respond to an
incomplete request for medical records. Health Insurance Portability and Accountability Act, 42 U.S.C. §§ 1320d-5, d-6; see also Acara v. Banks, 470 F.3d 569, 571 (5th Cir. 2006) (noting that there is no private right of action under HIPAA, and enforcement is limited to the Secretary of Health and Human Services).
Legal Disclaimer: I am not your attorney. The only method of establishing an attorney--client relationship with me is by a retainer and fee agreement signed by me, which is not offered via this site. The information given is not legal advice and is for general information only. My answer is limited to the facts presented and may not be complete or applicable to your situation. My answer is given upon the condition that you will contact an attorney licensed in your state to receive legal advice that applies to your particular situation.