They cannot restrict your right to travel just the children. Depending on if this is a CPS case or not you could have the case eventually transferred if you cannot move. Regardless continue to try to reestablish yourself to be the parent again to your five year old.
This is not a comprehensive answer and it is impossible to provide a meaningful response without a consultation. Call us for more information. 619.797.5456 www.mataelelaw.com