Assuming this is a criminal case, and your counsel was paid out of public funds, the court is within its discretion to deny you substitute counsel if it concludes there is no substantial basis for a new attorney. I'm also assuming you want new counsel, because it is well-established you have the constitutional right to represent yourself, however ill-advised that would be.
You have rights also however. You have the constitutional right to conflict-free, competent counsel. And "competent" is judged, not by who the attorney is, but his preparation, investigation, and legal analysis in your individual case. A common complaint may defendants have of their public defender or court-appointed attorney is lack of communication. And I'm of the belief that, even if the attorney does little else on the case, a thorough client interview and regular communication with the client is critically important to representation that will pass constitutional muster. In this regard, you also have the right to an on-the-record hearing regarding the issues you have with your attorney. Therefore, regarding your question about whether to appeal, that would depend on how the hearing went (if you had one), and the judge's reasoning.
If this is a criminal case, the court has a lot of leeway in preventing a person from representing themselves. In a civil case, which this appears to be, there may be less leeway and the court's reasons for denying this is important. One issue I can see is when someone is a defendant, the case has been going on for a while, and the court sees this as a delay tactic-they will liekly deny the motion.