Can I get the fees I paid to the lawyer back when I fire my lawyer?
It depends, what does your fee agreement say about those fees? If it was a refundable retainer that was an advance, sort of security deposit for fees for specific services that didn't get incurred, then yes, you should get a refund. If your fee agreement says the lawyer charges an hourly rate to be billed against the retainer, that's a conclusive indicator that the payment is an advance payment or a security deposit that is refundable unless fully earned.
But if it was a true non-refundable retainer, to ensure the lawyer's availability and not paid for the performance of any particular legal services, then no, because the lawyer earned that fee on receipt. Rule 3-700(D)(2) of the Rules of Professional Conduct. The State Bar requires that this fee not be "unconscionable" in light of the facts as they existed at the time the agreement was formed. Rule of Professonal Conduct 4-200 lists 11 factors to be examined in determining whether a fee is unconscionable, including (1) the relative sophistication of the lawyer and the client; (2) the amount of the fee in proportion to the value of the services rendered; and (3) the experience, reputation and ability of the lawyer. For example, requiring the client to pay a "minimum fee" on discharge of the lawyer was held unconscionable [In re: Scapa & Brown (1993) 2 Cal. State Bar Ct. Rptr. 635, 652]. For another example, a very experienced and well known criminal defense lawyer can justify this type of retainer even if their client is never indicted and the lawyer doesn't have to do anything.
There is also an issue of what the lawyer does with the money. Rule of Professional Conduct 4-100 provides that if the money is an advance, it has to go in the lawyer's client trust account, but if it's a "true" retainer, the lawyer can put the money in their proprietary account. a lawyer's failure to follow this rule can subject them to State Bar discipline as well as civil liability for professional negligence and breach of fiduciary duty.
Disclaimer: Please note that this answer does not constitute legal advice, and should not be relied on, since each state has different laws, each situation is fact specific, and it is impossible to evaluate a legal problem without a comprehensive consultation and review of all the facts and documents at issue. This answer does not create an attorney-client relationship.
Criminal Defense Attorney
In addition to my colleague's answer, your attorney is entitled to fair compensation for the services performed. If you paid five thousand dollars and the attorney put in ten hours of work valued at $300 per hour, you'd be entitled to get back two thousand.
Many criminal defense attorneys work on a flat fee basis, charging one amount for everything short of an actual jury trial, plus additional trial fees if the case actually goes to trial.
If the case gets resolved at an early stage, the attorney may get a lot of money without putting in a whole bunch of hours, but other cases wind up requiring so much work that the attorney is barely making minimum wage.
Under these circumstances, after deducting the value of the actual time spent on the case, there might not be anyting left to refund.