He may very well have a conflict, and he may need to withdraw. This is a separate issue from whether or not you are entitled to a refund of your retainer. You are entitled to a refund of any unearned portion of your retainer. the length of time this lawyer has been representing you, the number of appearances he has made in court on your behalf, and the number of hours he has soent working on your case are all factors. You may be entitled to a partial refund. If you want him to withdraw, request him to do so in writing, and request a refund of any unearned fee.
A client can fire their lawyer for any reason or no reason at all, (subject to court approval at times - especially if close to trial). Normally in Minnesota if you have a new lawyer the court won't have a problem with it. Asking for a refund of fees paid is a different issue. Firing your lawyer would not necessarily entitle a client to a partial refund in fees, nor would a possible conflict of interest. The written retainer agreement may have terms anticipating termination of representation, and how that would affect fees paid. If you haven't done so already, I'd suggest talking about it with the lawyer.