Can I keep my children's father from getting joint custody?

I am a single mother of 2 kids. The father has been around, but not actively involved in decisions with health, school, sports, etc. He helps financially "when he can" but not on a regular basis. He is now threatening to take the kids 50% of the time if I don't "GIVE HIM SOME"(can this be mental abuse?). My son, 11, has witnessed his fathers rage and does not want to live with him. He has only taken the kids (8 girl, 11 boy) overnight MAYBE 3 times in the past 6 years. What can I do? I don't trust him to take care of them for extended periods of time & we have never been married. I have left them with him for 3 days to go away on business & his mother ended up watching them majority of the time because he was BUSY....