You and your ex share the responsibility to make the custody and visitation plan happen. Often, when clients ask me the question, "Do I have to 'make' my child visit the other parent? I don't want to have to 'make' my child go if s/he doesn't want to." I ask them to replace the term "other parent" with either " pediatrician" or "math class." You can easily imagine kids not wanting to go to the doctor or to school but if your child had a problem with either of those would your first response be, "OK. Let me help you avoid that."? Of course not. You'd "make" them go because it's important for the CHILD. You would facilitate the conversation between the CHILD and the other parent so that any problems they have can be worked out between them. If the math teacher or the pediatrician wouldn't talk to you what would you do? You'd go over their head (to the head of the medical group or to the principal of the school) but you'd make sure it was taken care of. In this case that may mean you arrange with your ex to schedule an appointment with mediation at court or with a therapist to work out the issues but you don't fail to make the visit happen. Sometimes visits are boring, just like school and medical appointments. That doesn't change your obligation.
Will your ex charge you with contempt? Maybe. That's up to him and the burden of proof will be on him.