It's not the bias. It's the fact that he may have access to privileged attorney-client information that he may now use against you. Whether he were to do that deliberately or accidentally doesn't, I think matter. You should bring this to the attention of the lawyer and possibly by the court by sending him a written notice claiming the privilege.
This is something that a lawyer representing you would certainly consider and likely do. Or should I say, that's one of the things I would do. Each lawyer has some differences in how they approach problems in a case.