It is not per se unethical and it MAY or MAY NOT be to your advantage at trial--speak with your attorney about recusal and whether you might benefit from such a request.
NOT LEGAL ADVICE. FOR EDUCATION AND INFORMATION ONLY. Mr. Rafter is licensed to practice in the Commonwealth of Virginia and the US Federal Courts in Virginia. There is no implied or actual attorney-client relationship arising from this education exchange. You should speak with an attorney licensed in your state, to whom you have provided all the facts before you take steps that may impact your legal rights. Mr. Rafter is under no obligation to answer subsequent emails or phone calls related to this matter.
Sometimes asking a judge to recuse (withdraw from the case) backfires. The judge might refuse to recuse him/herself and then make things very uncomfortable for your lawyer. Going to the same bar does not mandate a recusal, though the appearance of impropriety could apply, depending upon the degree of contact they have with each other. If there is potential for the appearance of impropriety, the judge should generally recuse.