i have custody of my children, and plan on moving out of state in 7 months.The ex says no & I have no legal rights .

I feel I have rights due to the fact that there is a order of protection against him. He has also been arrested three times in the past year for breaking the order amongst other things. Iv'e heard I can take them and heard I can't. He has not paid any child support ever! he has been to rehab 5 times, and currently just left a half way house, he is also on a mandated drug treatment program for the next 5 months. Iv'e always worked, kept a house, and kept the kids in school. I have provided for them for the past 5 years while he has been doing his own thing. We just recently divorced, and in the agreement I have custody, he has visitation as agreedupon between us. With all of this said will I be able to move out of state and start a new life? He is going to fight till the end he says.