Just because a lawyer filed a Motion to Withdraw does not mean the court will grant that motion. The lawyer would have to give a reason why they are withdrawing and you will have a period of 15 days from the Motion Filing in which to Object.
Even though you have had multiple continuances, I would still ask for another so that you may obtain conflict free counsel. I doubt the judge will make you go to trial with a conflicted attorney who filed to withdraw or on your own.
Just because the attorney filed a Motion to withdraw does not mean he can do so automatically without the Court's approval. the Court can either deny it or grant it.
However, there are times (although mostly rare) that the court will grant the Motion to withdraw if the Court si satisfied that the client and attorney cannot work together to ensure effective assistance of counsel to the client. The Court will inquire into whether proper notice was given to client and whether client consents to the withdrawal.
The reasons for withdrawal are important which is not disclosed in your question. For example if the client intends to frustrate attorney's effective representation or the attorney's continued participation in the case would reasonably expose the attorney to ethical or professional violations or potential to exposure the judge may grant the motion to withdraw.
You have not disclosed your contribution to such act of withdrawal by your attorney. The court will also consider your acts or failures to act.Is there a breach of contract by you - for example failure to meet your financial obligations to the attorney (which is not the case based on what you have stated). There could be other breaches of contract issues.
Just because your attorney was in contact with your accuser does not automatically create a conflict of interest. Conflict of interest rules are pretty complicated. The nature, extent, and purpose of contact will be looked into to determine if a conflict of interest exists.
Most good attorneys do contact the accuser to find out facts and circumstances that may not be disclosed to the police or prosecution because sometimes the police or prosecution do not ask the appropriate questions of the accuser.
It is sad you are placed in a predicament. You can ask the lawyer for a refund through the court (you can try. It does not mean the court is inclined to intervene).
A good ethical lawyer would have given you sufficient notice to get new counsel. I would not place a client of mine in such a predicament when client has met his or her financial obligations per the fee agreement.
A bit of advise after the fact. The cheapest lawyer is not the best way to go. Also, just because the lawyer is close to your home or work is not always the best way to go for a client. Generally, these factors in the selection of a lawyer turn out for the worse for the client. Remember, it is your life or future at stake. You as a client must take responsibility in the proper selection of your lawyer. Further, just because the lawyer is well known or famous or very busy does not necessarily mean it is good for the client. or will get results for the client. A selection of a lawyer you must look to his or dedication to work hard and smart to meet your needs and expectation. I do that for each and every client. Finally, a lawyer who speaks openly and freely about your case(whether or not that is what you want to hear) and a lawyer who gives you the most skeptical view of your case may be better choice because it shows he or she is honest and will treat you right at all times.
Sadly, most clients ignore the how dedicated and ethical is the lawyer.