If you and your child's father only have an informal child custody agreement, then you may decide to stop visitations until a court order is issued. I do not recommend this however and would instead recommend that you move forward with your goal of getting a formal custody order. the critical issue is that you and your child's father need to establish a parenting plan and may possibly both need to take parenting classes. I would be concerned that your child's aggressive conduct may need to be moderated by an agreement to have him participate in age appropriate activities. Good luck with this.
This is not a substitute for a consultation with a family law attorney in your community.
If you have concerns about violence then you could contact CPS. A good choice would be to get the child into some play therapy to help curb the aggressive behavior. Since you don't have a formal parenting plan, you should try and talk to your ex and discuss the concerns with him. Taking your child without allowing visits and not involving CPS or the court, could backfire on you in mediation when he says you are withholding the child. Remember court is about providing evidence for what you are asking for. You should probably consult with an attorney to help you since the issues sound serious.
Answering this question does not form an attorney-client relationship in any way. The answer provided is for general information only as it is impossible to obtain the entire facts from the question posted here. This information should not be relied upon and is not intended as legal advice. You should always talk to a local attorney who can advise you on your specific matter.
Sign up to receive a 3-part series of useful information and advice about child custody law.