Generally client money is kept in a trust account unless the payment is compensation for work already performed. You should refer to your fee agreement on refund protocol, but normally unused portions are returned to the client.
Generally the retainer goes in escrow and the attorney bills against it for time charges until it is exhausted. New York rules prohibit non-refundable retainers. But most lawyers who have any business sense, in a hybrid billing arrangement, will ask for a retainer in an amount to cover a minimum of work that will be required. So don't expect to get a refund unless you discharge the lawyer early on or the case somehow settles early.
Any opinions stated in response to Avvo questions are based upon the facts stated in the question. Responses to Avvo questions are for general information purposes only, and should not be construed or relied upon as legal advice.
Most states require attorneys to put retainers in trust until they're earned; if you have a contingency for a successful case, though, the rest of that retainer is going to go towards the contingency if you're successful. In the event, however, that you want to dispute the attorney retaining any of the flat fee in the event of an unsuccessful case, you would need to be prepared for the attorney to provide you with an itemized accounting of time and expenses that may actually exceed the initial retainer - would you, in that situation, be willing to pay more?
Obviously, you should speak with a NY attorney about the issue - I can't speak to whether NY bans non-refundable retainers or how the remainder is handled or such.
No information you obtain from this answer is legal advice, nor is it intended to be. You should consult an attorney for individualized advice regarding your situation. No attorney-client relationship is formed by my responding to your question.
Unearned money you receive, like a retainer (as opposed to a flat fee) goes into escrow, often IOLA escrow. However, if your agreement states that the flat fee is the minimum fee, or is earned upon retention of lawyer, he can put that anywhere, including his sock drawer; he has earned it already, and is not billing you hourly.