You CAN sue a doctor for slander and libel. Whether you will win is another question - you will need to retain an attorney who can look at the actual statements that were made and determine whether you have a valid case.
The statement that you're taking "advantage" of your disabled client and using their disability to your advantage isn't actionable, because it's an opinion rather than a statement of fact, and it's not clear that the agency investigator could understand the doctor's statements to be defamatory --how are you supposed to be taking advantage of the patient?
In CA, slander (oral defamation) and libel (written defamation) have to be a PROVABLY FALSE statement of FACT to a 3rd party that damages the subject, and here, this doctor's statement seems to be an OPINION, and not a very specific one. If, e.g. the doctor had said, you've stolen $2K from the disabled person, or you're not a care provider with credentials from the XYZ nursing school, that would be actionable, since those could be specific, provably false facts. But stating that you're "lazy" or "don't earn the money you make" or are "a bad care giver" or that you "take advantage" are all opinions.
As for the doctor's betrayal of trust, the patient can choose another doctor if they want, and try to get one that will get involved in their housing issues and see them when they want to see them.
Disclaimer: Please note that this answer does not constitute legal advice, and should not be relied on, since each state has different laws, each situation is fact specific, and it is impossible to evaluate a legal problem without a comprehensive consultation and review of all the facts and documents at issue. This answer does not create an attorney-client relationship.
A roundup of the best tips and legal advice.