Yes, the Court of Appeal has the power to deny leave to file an appeal to vexatious litigants. The vexatious litigant statute covers proceedings in the Court of Appeal. The Presiding Justice will make his or her decision on the application for leave using the normal test found in CCP 391.7(b). If the appeal does not survive the application of the test, then leave will be denied.
It strikes me that a denial of leave would be an appealable order, since there will be no further proceedings in the Court of Appeal. Note that I have NOT researched the issue. Assuming, however, that the denial of leave is appealable, the remedy would be with the California Supreme Court.
You should be seeking the advice of counsel here. I'm not saying you should hire counsel to handle the appeal. I am saying you should hire counsel to advise you in as to the merits of your case, and to provide specific advice on occasion. If the Court of Appeal is unlikely to grant leave, it is better to know that now instead of wasting your time preparing an appeal that will go nowhere.
If you have been labeled a vexatious litigant, I would consult and retain legal counsel before you file any further legal proceedings. By definition, a vexatious litigant is a pro se litigant, i.e., one who is prosecuting a matter without an attorney. While an appellate court cannot prevent you from filing a notice of appeal with respect to any appealable order, it can sanction you for filing a frivolous appeal. Those sanctions can include sizeable monetary damages payable to the court and other parties for having to defend a frivolous appeal. You do yourself no good, and substantial harm, continuing to litigate a frivolous lawsuit.
I am licensed in California only and my response assumes California law. It is provided as general information only and is not legal advice. It must not be relied upon by you. Legal advice must be based on the exact facts of the particular situation. This forum does not allow for the discussion of those specific, exact facts. I provide legal advice only during the course of an attorney-client relationship. The exchange of information through this forum does not establish such a relationship with me. That relationship is established only by personal and direct consultation with me followed by the execution of a written attorney-client agreement by each of us. In this, as in all cases, you should personally consult with an attorney in your area for legal advice.