it seems like my public defender is not even on my side
Divorce / Separation Lawyer
If you fire your PD, you will not be appointed another one. You can fire your PD but you had better be prepared to retain private counsel.
General Practice Lawyer
Just tell your public defender that you want him or her to withdraw from your case but like Ms. Goldstein said, if you fire your PD you won't get another one.
Nothing I say while answering your question forms an attorney-client relationship. You should hire an experienced attorney to answer your specific legal questions.
I would encourage you to communicate your frustrations to your public defender. Tell he or she exactly what you wrote above, and see how he or she responds. A lot of the time, they are very busy, but good at what they do. If you don't get the attention and advocacy you expect, let the judge know. No guaranty you will get a new lawyer though unless you have a serious conflict. Good luck.
This response is not intended to provide legal advice or to create any attorney/client relationship.