Almost all attorneys will use a contract for legal services. Refer to your contract to see the terms of the contract and whether the fee is refundable and under what circumstances. There are reasons for an attorney to withdraw from a case even though the fee may have been paid. (ie...course of action by client that is unthical or illegal or if you create a conflict between you and he).
If there is no contract, then he may be in aposition that he has to defend his fee by showing that he earned it by the number of hours involved.