APS is a branch of government and it has vast power. It can initiate guardianship proceedings and it can refer for criminal prosecution. You would be best served by consulting an attorney experienced in dealing with them. They know they can get away with bullying you and they will.
DISCLAIMER: This is not specific legal advice and does not establish an attorney/client relationship.
Chances are that you don't know what evidence APS is looking at. It may well look like neglect or abuse to them. At this point you should hire an elder law attorney. You don't want to wait so long that you need a criminal defense attorney, too.