If you don't have sole custody already, you should file for a modification so the judge grants you sole custody. You shouldn't file to terminate his rights because then you will lose the right to receive child support. Even though he hasn't been paying, the fact that he has a warrant out for his arrest may cause him to eventually pay you.
THESE COMMENTS ARE NOT LEGAL ADVICE. They are provided for informational purposes only. Actual legal advice can only be provided after consultation by an attorney licensed in your jurisdiction. Answering this question does not create an attorney-client relationship or otherwise require further consultation.
Sign up to receive a 5-part series of useful information and advice about child custody law.