If you can show "good cause" to the court, you are likely to have your court appointed attorney replaced. Best of luck.
The author of this answer is an Attorney-at-Law, licensed to practice law only in the state of Arizona. Unless both you and the author have signed a formal retainer agreement, you are not the author's client, and the author's discussion of issues does not constitute legal advice. Opinions expressed herein are solely those of the author, and are neither privileged nor confidential.
If your boyfriend has a complaint about the attorney the court has appointed to him, his recourse (not yours) is to go to the court and list his complaints. It is unlikely to result in a change in court appointed attorney unless there is "good cause" shown that shows his case is being hurt by the representation style of his court appointed attorney.
I am concerned about what the matter he has is regarding. I think most would agree that it is inneffective for a court appointed attorney on a major felony to not communicate with his client. A court appointed attorney may choose to speak to a client via the telephone or via a video monitor, if he is unable to visit a client in jail as often as possible. Given the fee structure for a court appointed attorney, they often put in much more work than they are paid for. I doubt very much that it is the court appointed attorney being "afraid to visit his client." Instead, that attorney is likely so overworked and underpaid on the case that it is difficult when the client wants to see him and instead he or she will do it when they are going to visit other clients as well. If this is an appeal or a PCRA, it becomes even more complicated since that attorney may need to drive across the state, spend money on gas, rent a hotel and never get reimbursed by the court.
This is not intended as individual legal advice and there is no attorney client relationship established by this answer. It is advisable that you seek individualized legal assistance. This is not a substitute for hiring an attorney.