You should consult a family lawyer. You can start a custody action to have the court determine what schedule would be in the best interest. There are many factors the court will consider when determining where father can exercise his custody. The age of the children, father's past behaviors around the children and the condition and safety and cleanliness to the home where the children will spending time with father will all be considered by the court in determining an appropriate custody schedule for father.
At this point, without a Court Order, you and the father are on equal footing. If you allow him to take the children he is not required to return them unless you take him to Court. Continue to insist that he only visit with the children in your presence until one of you files for custody. Once you have a Court Order, you will not need to restrict his access to the children. One of you should file a Custody Complaint in the County in which you reside with the children.
Sign up to receive a 5-part series of useful information and advice about child custody law.