In Georgia, the age of consent is 16 years. So, a 15 year old would be a no no (unless they are married). Your son's father could indeed get in trouble. He could be prosecuted for statutory rape under Georgia law, OCGA 16-6-3. If he is still 20 years old, and she is still 15, this is a felony with a 1-20 year range. Once he is 21, and if she is still 15, this becomes a 10 year minimum, with a 20 year maximum. Now, you use the word "girlfriend". That does not strictly mean they are having sex. What I say above applies if it can be proven that they have had sexual intercourse. If they have done other sexual acts, the crime may be child molestation.
Neither you nor your son should get into any trouble. However, it may not reflect well upon you should you still allow your son to spend time with his father knowing what is going on over there. Terminating parental rights has a high hurdle associated with it. I am not advising that his rights will or will not be terminated based on these facts. You have suspicions which should lead you to seek the counsel of a family law attorney who also is familiar with criminal law. You should immediately take steps to address these concerns.
Call the police. If he has sex with the 15 year old, he is going to prison for statuatory rape and will be a registered sex offender for life. Even if he does not have sex, he may be guilty of other serious crimes. His behavior is not only sick, but a felony, and you will help get a child molester off the streets (and protect your child).
That police report and police investigation will also likely help document a civil case, for which you will hire a lawyer, to attempt to limit or change his visitation. You definitely need a lawyer.
If you find this answer helpful, please mark it here on AVVO as helpful. In answering you, I am attempting to communicate general legal information and am not representing you (and am not your lawyer). Do feel free to call me at 404-768-3509 if you wish to discuss actual representation (the phone call also does not retain counsel; that requires an office visit and appropriate paperwork). In that a forum such as this provides me with limited details and doesn't allow me to review details and documents, it is possible that answers here, while meant to be helpful, may in some cases not be complete or accurate, and I highly recommend that you retain legal counsel rather than rely on the answers here. (You can also email my office at firstname.lastname@example.org . An email also does not retain my office, but can help you get an appointment set if you prefer not to call). I am happy to discuss possible representation with you. Any information in this communication is for discussion purposes only, and is not offered as legal advice. There is no right to rely on the information contained in this communication and no attorney-client relationship is formed. Nothing in my answer should be considered as tax-advice. To ensure compliance with IRS Circular 230, any U.S. federal tax advice provided in this communication is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer (i) for the purpose of avoiding tax penalties that may be imposed on the recipient or any other taxpayer, or (ii) in promoting, marketing or recommending to another party a partnership or other entity, investment plan, arrangement or other transaction addressed herein. Note that I am only licensed in Georgia and thus cannot practice in other states. I am also required to advise you, if your question concerns bankruptcy, that the U.S. Congress has designated Ashman Law Office as a debt relief agency that can help people file bankruptcy.
From the way you've written your post, it sounds as if the 20 year old father is the biological father but perhaps not the legal father? Even if he filed an acknowledgment of paternity and legitimation, he has no right to custody or parenting time with your child absent a judicial determination. Here is a relevant document published by the DHS: http://dhs.georgia.gov/sites/dcss.dhs.georgia.gov/files/imported/DHR-OCSE/DHR-OCSE_Paternity_Establishment/Paternity&LegitimationOfChildBenchCard-FINAL.pdf
This answer is offered for informational purposes only. It is not offered as nor does it constitute legal advice. This answer does not constitute an attorney-client relationship. Do not rely on this answer in prosecuting or defending against any criminal or civil legal action. Speak to an attorney in your area about how to protect yourself and your interests.