Generally when professional services are rendered, there is no promise nor reasonable expectation that they be wholly error-free or else be free. You owe your accountant either what you specifically agreed upon, or what was your prior course of dealing, or at the very least restitution for what was done.
That being said, if the errors were below the standard of care for CPAs and the errors were the proximate cause of your damages, then perhaps you have a claim for negligence. However, the viability of a claim like that will depend a lot on the specific facts and also the damages.
You may also consider that while your CPA declined your request that the fee be waived, he may not be interested in trying to sue you for the fee, especially if it was a small one. Whether or not then you choose to pay, may be more a matter of personal choice and ethics than one of legal ps and qs.
No legal representation exists by virtue of this answer. Consult your attorney. Licensed to practice law in Indiana and Illinois. Circular 230 Disclosure: any U.S. tax advice contained in this communication is not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code, or (ii) promoting, marketing or recommending to another party any matters addressed herein.
Must you pay? No. But if you do not pay, will the accountant sue you? Maybe. Because, as the other attorney pointed out, there is no guaranty of error-free work when you hire a professional. On the other hand, why would the accountant want to sue you when the accountant made mistakes which cost you time and money.
If the CPA corrected the returns and you got back money, the question is: what damage did you suffer? Your lost time is not going to be compensated by a court. You lost no interest because if you got a refund you probably got interest from the government. So I don't know what "significant...money" it cost you. You are probably best off simply leaving that accountant and finding a new one who is highly recommended by your friends, neighbors or business colleagues and forgetting this episode as a bad experience.