Sounds like you have a workable but dysfunctional custody arrangement. before you do anything, speak with a family law attorney in detail about the pros and cons of trying to address the situation in court.
You should probably consult a family law attorney. You state that your son loves his father and his father appears fairly involved in his life, however, you son is often afraid of him. Is your son in therapy? What is the father doing to make him scared? If you son is afraid of his father, but will not give you details, then it would make sense for someone trained in child psychology to be meeting with your son to possibly find out why he has these fears and possibly if they are legit. If your agreement provided for joint decisions on therapy and your ex will not agree to it, then you will likely need to file a motion and ask the court to order it.
Sign up to receive a 5-part series of useful information and advice about child custody law.