This depends on the type of retainer agreement (contract) you signed with the attorney. If the case required a retainer, and the retainer is nonrefundable, then the attorney earns said fee upon receipt and it is nonrefundable as long as the attorney performed and earned a reasonable amount of said fee. If it was a flat fee, then it is likewise earned at receipt.
If you and the attorney have a disagreement and it cannot be resolved, then the lawyer may keep the nonrefundable retainer and seek a withdrawal from the case since the client and lawyer are disagreeing.
Also note, that once a nonrefundable retainer runs out (i.e. billed out at the attorneys hourly rate), then the attorney will likely begin billing you per their hourly rate.
Take a moment to reread your retainer agreement.
Note: The information presented at this site should not be construed to be formal legal advice nor the formation of a lawyer/client relationship.