If you have not done so already, I respectfully suggest you discuss your concerns with your public defender. Perhaps, you have just been miscommunicating.
If you have already spoken with your public defender regarding this issue and it has not been resolved, you should advise him/her you would like new counsel. Your public defender should then advise you as to your options. Some public defender's offices will allow your attorney to withdraw and will then appoint another public defender or court-appointed attorney to represent you. Other public defender's offices will tell you, you only get one public defender; and if you fire your public defender, you must retain counsel on your own or represent yourself.
If your relationship with your attorney has deteriorated to a point where you do not wish to speak with him/her, I suggest you contact the public defender's office directly. Again, you may not have the option of getting another public defender. The 6th Amendment right to counsel does not necessarily give you the right to counsel of your choice.
Pursuant to the U.S. Supreme Court's decision in Gideon v. Wainright, even if you cannot afford an attorney, you can have one appointed to represent you, free of charge.
That being said, you are NOT entitled to a "free" attorney of your choice.
I agree that you should try to speak with your current public defender and try to work things out. Please understand that you are not the public defender's only client, and that he/she is most likely doing the best job he/she can do under the circumstances. Contrary to some perceptions, public defenders do not "work with" the prosecutors or courts to try to make things hard on their clients; public defenders are licensed attorneys who, like all attorneys, have taken oaths to represent their clients to the best of their abilities.
If you are still not satisfied, I would suggest that you contact any and all family members or friends to assist you in raising enough money to hire an attorney of your choice. If you decide to go this route, try to do it as early in your case as possible.