Three year olds can be pretty clingy in the best circumstances. You should hear my daughter when we have to leave the playground at day care to go home. If these are short supervised visits, I would not worry about your child's safety during visits even if he does cry. But you do have a problem here: your ex is not obeying the court order which says he must contact and work with the counselor, and he's not obeying the court order when he repeatedly stands you up at the visitation location.
It would be appropriate for you to bring this back before the judge. You could ask for less frequent visits. You could ask for more structured communication about visits so it is not so inconvenient (for example, he has to call the day before and/or the morning of the visit.) The judge could ask your ex why he should not be held in contempt of court for refusing to work with the counselor. A lawyer could write a motion for you and represent you in court. There may be a walk-in clinic or help desk at the courthouse where you can get help if you cannot hire a lawyer.
It sounds like you may want your ex's parental rights to be terminated. That is not going to be easy. Most of the time, even when a parent does bad things like missing visitation, it is better for the child to have some kind of relationship with both parents.