The appointed representative should not be allowed to collect a fee if he was not successful in securing you benefits. I would recommend requesting a letter in wiring from him withdrawing from your claim and waiving all fees and costs related to your representation. If he refuses, send a letter to the current judge advising that you have requested the attorney to withdraw and that you do not feel he has earned his contingency fee for securing benefits.
Attorney Bruni has given you good advice. I'll just add that IF the prior rep were approved by SSA for any fee then you will get a notice and you will have the opportunity to object to the payment of a fee. You'll want to carefully look at all documents you might receive from SSA.
It is impossible to give specific answers to questions without meeting and fully discussing all of the potential issues that may not be addressed by your question. The answer(s) provided should be considered general information and are not legal advice. Only after a thorough personal consultation could specific legal advice be given. Viewing this information is not intended to create, and does not constitute, an attorney-client relationship. To enter such a relationship you and I would need to consult in person and form a mutually agreeable written contract of engagement. The answer(s) provided in this forum is intended to educate you and point to issues for you to raise in a consultation with a lawyer of your choosing who is appropriately competent in the field of law that your question concerns and who is duly licensed to practice in the jurisdiction where you live and/or where the events giving rise to your question occurred. You should not take any action that might affect your claim(s) without first seeking the professional opinion of a licensed attorney. There are often strict deadlines for filing suit, responding to a suit or making an appeal and you need to personally consult with an attorney to make sure that you understand and meet those deadlines.
If you ultimately prevail, this lawyer may nevertheless file a fee petition in your case. The fee agreement will no longer be valid. You should ask him to withdraw and "waive" any fee. If he refuses, then file a Bar Complaint with the State Bar in your state. If as you assert he did nothing, he should do the ethical thing. He should withdraw and waive any fee. Of course, if you prevail, you can also object to his fee petition.
Remember to designate a best answer.
This response is meant to be information only and should not be considered to be legal advice. This information is not meant and should not be construed to be the formation of an attorney client relationship. I practice Virginia Workers compensation law and Social Security Disability law.