If you already have a court order then you both need to abide by it. If he is not exercising his visitation like he is supposed to, then you can take him back to court and get a judge to limit it. However, simply refusing visitation yourself makes you as big an offender as him. He might hire a lawyer and try to have you held in contempt of court. However, if he has no money and cannot afford a lawyer, then self-help on your part will probably not be much of a risk. The "cops" will not likely come to his aid unless and until he has a court order specifically directing them to do so.
If I understand the facts correctly, you and the father of your children are subject to a Court Order which grants the two of you joint legal custody of your children. The Order also grants you primary physical custody and the father specified visitation with the children. Since last summer, the father hasn't exercised his scheduled visitation on a regular basis. Recently, when he did exercise visitation, the father refused to transport the children back to you at the end of the visitation period. Finally, he is demanding to exercise visitation this weekend, even though it isn't his scheduled weekend. When you told him it wasn't his weekend, he threatened to call the cops.
Firs,t you should locate your Child Custody Order and check to see exactly what it says about the visitation. If by its terms the Custody Order clearly shows that this coming weekend is your weekend, then you do not have to let the Father exercise visitation. Just be sure to have the Court Order on hand so you can show it to law enforcement if a patrol person shows up at your front door.
Often I hear from custodial parents like you who have problems with the non-residential parent when it comes to him exercising his visitation rights. The Court cannot make the father exercise every scheduled visit, but if he continues to not show up for visits, you can take him back to court and ask the Judge to modify the visitation schedule so that it more closely mirrors the visitation pattern the father has been exercising. If the Court agrees with you, then at least you would have a bit more predictable schedule for your own planning purposes.
Sign up to receive a 3-part series of useful information and advice about child custody law.