You did not know what he was like before you bore his child? Just keep copies of the texts and try not to make this an issue, he probably leaves the child with hi mother anyways. and ask the court to order him to give you his phone number! If you remove her, who would you substitute? Get real, he is a bully and angry, let the little boy exhaust himself and maybe he will leave you alone.
My name is Stephen R. Cohen and have practiced since 1974. I practice in Los Angeles and Orange County, CA. These answers do not create an attorney client relationship. My answers may offend I believe in telling the truth, I use common sense as well as the law. Other state's laws may differ.. There are a lot of really good attorneys on this site, I will do limited appearances which are preparation of court documents it is , less expensive. However generally I believe an attorney is better than none.
I am assuming that there are supervised visits because of the existing restraining order. What puzzles me though is the fact that you seem to be using your mother to "supervise" your ex's mother on visitation. Absent some specific language to the contrary in a court order, I do not believe this is permissible even for supervised visitation. Moreover, if you are insisting that your mother be there to "monitor" visitations, YOU could in fact be violating a court order.
If your primary concern is the environment and/or your child's unfamiliarity with either his father or paternal grandmother, then I would suggest going to court to seek professionally monitored visitation. This would eliminate the issue of the paternal grandmother and would also ensure a safe environment for your child to interact with her father. I would suggest contacting an experienced family law attorney to discuss your options further.