You can ask the court to modify the parenting plan to eliminate his visits, putting the burden on him to go to the court if he wanted to resume visits at some point in the future. He would still have to pay child support, but he would have no right to demand the child.
You would need to make sure the child's daycare, school, etc., had a copy of the court order so he couldn't just show up at those places too.
Regarding what happens if you aren't in the picture, you need to sit down with an estate planning attorney to draft the documents necessary for a probate court to know what your wishes are. You also need to consult with family. Most likely, family would have to file a non-parental custody petition if you were to die. Given the father's crummy history, it's unlikely any court would place the child with him.
Sign up to receive a 3-part series of useful information and advice about child custody law.