Unless you have a custody order, or a restraining order, you have no basis to keep him from having contact with the children. Whether a judge will view his conviction as a reason to keep him from having contact with the kids is impossible to say without more information:
How long ago was he convicted?
What, SPECIFICALLY was he convicted for doing?
Has he ever been convicted of anything else? What and when?
Assuming that he did whatever it was he was convicted of doing, what does that tell us about his ability to act appropriately as a parent?