No, you absolutely do not have the right to withhold contact between your children and ex-husband. If he is being abusive and showing up unannounced at your home at times when he is not scheduled to pick up the children, you can contact your local police department and/or refuse to allow him in the home. If your ex-husband is not complying with the present Court Order, you have legal recourse and can file a Motion for Contempt.
Your children need both of their parents and your ex-husband has been awarded joint custody of your children, that includes the right to be a part of all major decisions involving your children as well as the right to exercise his timesharing.
Furthermore, if you wish to relocate to Georgia, there is a statutory procedure that must be followed, including the filing of a Petition to Relocate and allowing your ex-husband an opportunity to object.
Good Luck in working through these issues!
Sign up to receive a 3-part series of useful information and advice about child custody law.