Has he been adjudicated the father in court? If not, you have sole custody and placement and there aren't any rights to remove.
In accordance with the Avvo community guidelines, this communication does not constitute "legal advice", nor does it form an attorney-client relationship. You should seek counsel in your geographic area regarding any specific questions.
If father has been adjudicated the father by the court, depending on the specific facts of your case you could file a voluntary or involuntary termination of parental rights action. However, your chances of success are significantly reduced if there is no one is assuming the terminated parents rights (example: stepparent adoption). Many courts still hold to the view that a uninterested parent is better than no parent. At a minimum, the uninterested parent remains obligated to financially support the child. Hope this helps.
My answer does not automatically make me your attorney or create a former-client/current-client relationship with you. I encourage you to speak with an attorney before acting upon any of the statements I make. If you would like to speak with me further about your issue, please schedule a free initial consultation (in person or over the phone) by calling my office at (920) 459-8490, or by emailing me at firstname.lastname@example.org. You can also visit www.klpplaw.com for more information.
One thing to ask yourself is why you would want him removed as a custodial parent? You stated that he doesn't pay child support, but why not? Has an order been put into place requiring him to pay support? If so, why would you not want to enforce the order? Terminating a parent's rights, even a vacant parent, is no easy task and the courts are wont to do it. Are you afraid that one day in the future he will show up and demand placement time? Your best bet is probably to attempt to establish and enforce a child support order rather than a TPR, but for either scenario you will need to see an attorney to assist you. Unless the father has abused the child or performed some conduct that would endanger the child?