You can live anywhere you want. A court has the authority to decide where the child lives.
If you leave, the father can either do nothing, or file for custody/visitation. For the next six months after you leave, Virginia courts, most likely Prince William JDR court, will have jurisdiction to decide custody (and child support).
If the father does nothing for six months, then the home state jurisdiction changes to where the child is. Now, father has to litigate out of state.
Though I do not practice in VA, I have some thoughts that may provide a useful answer.
In CA, you could leave not only the home but the state. However, you wold need to undertake certain measures rather rapidly. Among them would be to file re custody and to promptly notify father of child's whereabouts. Though VA may to some extent have at least initial jurisdiction, I believe that there is an interstate compact [which VA may or may not belong to] that requires either court to decide placement of kid and, fairly quickly to decide what court has jurisdiction.
Many cities have shelters for abused women and their children. You probably should seek some assistance with them.
As to Dad being aboe to do much about contesting where the chile lives, based on your post, I would think it unreasonable that some court would give him physical custody.
The above is not intended as legal advice. The response does not constitute the creation of an attorney client relationship as this forum does not provide for a confidential communication.