I understand your facts but you did not ask a question. If the current order states that you are restricted to a particular county or state, then you must ask for relief from the court to move. If there are no restrictions, then you are free to move.
If there are no provisions restricting you to a county or state and he files a modification, then you need to hire an attorney to assist with defending this modification.
The information provided is not advice but a legal perspective and you should schedule a consultation with the lawyer of your choice.
Relocation of the child(ren) after an award of custody usually require consent of the court. If there is a visitation order, there was a custody award to you as custodial parent. Regardless of the fact that he has not taken advantage of is visitation rights, you cannot unilaterally remove the child.
Get yourself a lawyer to help you petition the court for a right to relocate; then let your ex try to convince the court that he suddenly has dire need to visit with the child(ren).
This answer is provided for informational purposes only and it is not intended as legal advice. Additionally, this answer does not create an attorney-client relationship. If you wish to obtain legal advice specific to your case, please consult with a local attorney