I would suggest talking to your lawyer about your concern. Fee agreements vary from lawyers to lawyer, and although there is a fee dispute board, most lawyers will talk to you and treat you fairly. A lot of time, much of what your attorney does is behind the scenes and you don't always realize what all they have been doing.
Mr. Shaffer makes a good point. I would add a suggestion, which is to check your written retainer agreement (if you have one.) My retainer agreement expressly states that the client may be entitled to a full or partial refund if the relationship does not work out. Your agreement may state the same, in which case it might put your mind at ease before talking to your lawyer. Best of luck to you.
Please be advised that this comment should in no way be interpreted as legal advice, nor should one rely on this comment in making a final decision about his or her case. Each case is fact specific and peculiar in its own way. Thus, you should consult with an attorney in-person or over the phone, explaining your particular situation, so that the attorney can render fully effective and competent assistance tailored to your particular situation.