An attorney and client have a business relationship. If you are done with that attorney, fire them and move on pro se. I would certainly go to their office and have a meeting first. I'm sorry you are experiencing problems.
When you retained the attorney you should have been provide your Fee structure in writing. Most lawyers charge an hourly rate for their services versus a flat fee. I do not believe simply walking away is the correct action. You likely entered into a contractual agreement for legal services and simply leaving without paying your bill may be a breach of that agreement.
Most local bar associations have a "Fee Dispute Committee" which you may utilize to moderate your fee dispute.
It is highly probable that your fees incurred to date are appropriate, but you have not provided enough facts to appropriately assess them. You should receive an itemized bill regularly, if not monthly, of each task performed by your attorney.
I suggest calling the attorney. Perhaps a simple mistake in billing occurred.
Sign up to receive a 10-part series of useful information and legal advice about the divorce process.