The issue is not conflict of interest, the issue is potential "bias." A conflict of interest is more to do with the judge being the opponents relative, or something along those lines. (i.e. the judge having some potential personal stake in the outcome of the case).
Bias, under the situation you describe, is a pretty flimsy argument. Problem being, you are questioning the judge's credibility to judge your domestic issue dispassionately.
You can try, but I think you would lose (or be forced to appeal, and then lose).
Instead of trying to get the judge removed for bias or cause, you can ask for a substitution of judge. This can be done without reason but there are time limits in which you must ask for the substitution.
Legal disclaimer: No attorney-client relationship is formed by this communication. Any recommendation/information given through Avvo should be considered only as opinion.
I think that asking for the judge to recuse himself would probably be counterproductive. You might try requesting a substitution of judge if the case is early in the process, though. Or, just remember that judges deal with thousands of people each year with criminal problems and family problems and other problems and you are probably just one drop in the bucket. If he does remember you and your cases were unremarkable, the judge at most probably remembers the general gist of what happened in your cases and maybe what appeared to have been the root causes (i.e. drug or alcohol use and/or mental health challenges.) In a small county, you should also know that replacing one judge with another might not be terribly effective because the court staff and other judge might end up chatting with each other about the case or the judge might think he recognizes you from the courthouse and pull your other court files. In an adversarial system like ours, the opposing party usually does a pretty good job of pointing out the flaws in the other party, too. Get an attorney if you have something worth fighting about (i.e. placement or property division).
Please mark answers you appreciate with positive feedback!<p><a href="http://www.msm-law.com/nicholaspasse.html">Attorney Nicholas J. Passe</a><p><l>Disclaimer: Per the avvo.com community guidelines, no attorney/client relationship is created by the asking or answering of questions on this web site, nor do the answers constitute legal advice. Always hire an attorney before making any important legal decisions. Posting details of a case on avvo.com may be subject to discovery in criminal or civil litigation, so erring on the side of nondisclosure is wise.<p><a href="http://www.msm-law.com">Moen Sheehan Meyer, Ltd.</a>