The answer is in the details of your case. It depends on for how long this has been happening, the inconvenience factor for you and the affect on your child.
Visitation is a right that a parent holds but not something that the court will force him/her to exercise. It typically will stand, even if not exercised, unless the other party can show some type of harm, inconvenience, hurt to the child, etc.
Is there a reason you want to take it away? Is it just that you don't want to deal with the wishy washy behavior? Or is there something more?
This is not legal advice and is not intended to create an attorney-client relationship. You should speak to an attorney for further information.
Depends on how long he has been blowing off his visitation. My suggestion would be to first seek a stipulation that he needs to call ahead a half hour before said visitation to confirm that he is serious and on route.
You can ask the Court to modify the parenting time but not knowing all of the facts and circumstances of your case would make it difficult to predict how receptive the judge would be to it. You should make an appointment with counsel to map out a strategy.
Sign up to receive a 3-part series of useful information and advice about child custody law.