It may or may not have any bearing on a child custody case. The courts really look at each situation. What kind of crimes were they? When did they occur? Was there domestic violence involving the child? Sex offenses? Drugs? Also, the judge is given broad discretion in custody situations. Remember, the best interest of the child is the axiom.
The court will address the allegations, generally in mediation first, to see if an agreement can be reached between the parties. If either party does not agree with the mediator's recommendation, they can take the case to trial. A criminal background may have no affect on a custody case, or it could be determined that it is not in the best interest of the child to be with the parent with the record, depending on the circumstances.
What sort of criminal record? How long ago was it? Does the mother have a criminal record? Depending on the specific facts and circumstances, it may or may not affect custody. Also, how long has the mother had sole custody? Judges oftentimes like to maintain status quo in the best interests of the children. A judge will not disturb a long-standing and effective final custody order unless there is documented change of circumstances.
It may have a big affect or none at all. Some of the issues that the Court will address will have to be the nature of the crime, when the crime/conviction occurred, and whether the nature of the crime will affect the health, safety, and welfare of the minor children in the future. For example, if a father has a domestic violence conviction, that may keep him from having custody of the children. However, other minor issues, or issues that happened long ago, may not be given much weight by the court.