I advise that you hire a family law attorney as soon as reasonably possible. Without opening this up to a back-and-forth discussion of your facts, it is not clear if you and the father are divorced or if paternity, legitimation, etc., is all that has occurred. Either way, it sounds as if there was already a court action providing for joint legal custody. In that event, you could initiate an action in juvenile court to attempt to terminate the rights of the father. Keep in mind that termination of rights, per Ga statutes, requires a showing that by doing so, the child would not be left destitute, lacking in food, clothing, proper shelter, etc.. Based on the facts presented herein, it looks like you may (though I'm not saying it's certain) have a sufficient basis for seeking such.
I'll bet the Georgia statutes provide grounds for an involuntary termination of parental rights. Texas has such provisions. Find an experienced family law attorney in Augusta and get in to discuss the situation immediately. While he is in jail is a great time to do it. The fact that he is a convicted felon, has battery charges pending, and your child is seeing a counselor because of the issues should work to either terminate his rights or severely limit his access and that of his family. If his dad is making such statements, tell the attorney. The grandfather needs to pulled up short.
I am not intending this to be legal advice, because I don't know the particulars of your situation. Call me if you would like to discuss this or other isues.
You should speak with a qualified family law attorney. A court will not terminate his legal rights simply because you don't want to have to deal with him; however, there are circumstances in which parental rights can be terminated. Review Article 2 of Chapter 11 of Title 15 of the Georgia Code (linked below).
DISCLAIMER: This answer does not constitute legal advice and no attorney client relationsip has been, or will be, created until a valid engagement agreement is signed. No duty arises from this posting. Answers posted here are general and made with limited knowledge of the actual facts of your case. Always speak with an attorney licensed to practice law in your jurisdiction if you wish legal advice specific to your case.
Courts rarely terminate rights as that also eliminates support, but you may have grounds to change custody and visitation. Meet with a good family law attorney in your area to determine the best strategy.
If you find this answer helpful, please mark it here on AVVO as helpful. In answering you, I am attempting to communicate general legal information and am not representing you. Do feel free to call me at 404-768-3509 if you wish to discuss actual representation (the phone call also does not retain counsel; that requires an office visit and appropriate paperwork). In that a forum such as this provides me with limited details and doesn't allow me to review details and documents, it is possible that answers here, while meant to be helpful, may in some cases not be complete or accurate, and I highly recommend that you retain legal counsel rather than rely on the answers here. (You can also email my office at email@example.com . An email also does not retain my office, but can help you get an appointment set if you prefer not to call). Any information in this communication is for discussion purposes only, and is not offered as legal advice. There is no right to rely on the information contained in this communication and no attorney-client relationship is formed. Nothing in my answer should be considered as tax-advice. To ensure compliance with IRS Circular 230, any U.S. federal tax advice provided in this communication is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer (i) for the purpose of avoiding tax penalties that may be imposed on the recipient or any other taxpayer, or (ii) in promoting, marketing or recommending to another party a partnership or other entity, investment plan, arrangement or other transaction addressed herein. I am also required to advise you, if your question concerns bankruptcy, that the U.S. Congress has designated Ashman Law Office as a debt relief agency that can help people file bankruptcy.