You "get out" by resigning. You send a letter of resignation to your friend and anyone you have been doing business with on his behalf. If the POA form was properly set up, then there would be an alternate or successor agent named in the document, who can take over, upon your resignation. If this was not done, your resignation will leave your friend in a vulnerable position. Whether that impacts your decision or not is not clear.
***Please be sure to mark if you find the answer "helpful" or a "best" answer. Thank you! I hope this helps. ***************************************** LEGAL DISCLAIMER I am licensed to practice law in the State of Michigan and have offices in Wayne and Ingham Counties. My practice is focused in the areas of estate planning and probate administration. I am ethically required to state that the above answer does not create an attorney/client relationship. These responses should be considered general legal education and are intended to provide general information about the question asked. Frequently, the question does not include important facts that, if known, could significantly change the answer. Information provided on this site should not be used as a substitute for competent legal advice from a licensed attorney that practices in your state. The law changes frequently and varies from state to state. If I refer to your state's laws, you should not rely on what I say; I just did a quick Internet search and found something that looked relevant that I hoped you would find helpful. You should verify and confirm any information provided with an attorney licensed in your state. I hope you our answer helpful!
Look to the Power of Attorney document to see if it names an alternate agent. You cannot be forced to act as someone's agent, however, if your friend had dementia and a difficult family situation I recommend you consider having a Conservator appointed to help him handle his finances. Sending him a letter explaining that you are stepping down as his agent will certainly get you off the hook but it may end up putting your friend in dire straits. I recommend you schedule a consultation with a local elder law attorney.
DISCLAIMER: The information contained in this answer is based on Oregon law and is subject to change. It should be used for general purposes only and should not be construed as specific legal advice by Fitzwater Meyer Hollis & Marmion, LLP, or its attorneys. Neither this answer nor use of its information creates an attorney-client relationship. If you have specific legal questions, consult with your own attorney or call us for an appointment.