do I have the right to refuse him changing his mind in the furture or at his every whim?
This is not a good plan. Even in the best scenario where he waives all future visitation, he can always demand future visitation.
Here is a better plan. File a motion to modify the parent child relationship. Sign an agreed order giving him VERY LITTLE visitation, say the first weekend of every month. You can also put in a provision that if he is more than one hour late to pick up the kids, you are allowed to exercise alternate plans for them, that way you're not sitting around waiting for him all day on his allotted days. If he ever wants more, he'll have to sue you for it, which he may not do.
But what's his game plan here? If he doesn't want to see the kids, he doesn't have to even under the existing order. Is he trying to build a case that he shouldn't have to pay child support? Those are two totally different rights and obligations.
Just based on your question, I think something odd is going on here and I think you should consult with a local family law attorney. Give the attorney all the facts in your case and get some advice carefully tailored to your specific circumstances. Don't be afraid to pay a consultation fee--a couple hundred dollars will save you thousands of dollars in future trouble.