If the question is can you file, the answer is yes. If the question is what would your likelihood of success be if you filed for sole custody, the answer is "it depends." Obviously the facts you have presented are compelling, but your paragraph is necessarily limited and I am sure that there is plenty more you could share about your situation.
One thing you should be aware of is that California generally favors parents remaining in contact with their children. If your goal is to keep him from seeing your children at all, it would be very difficult -- if not impossible. Depending on your individual facts, circumstances, and goals, an attorney could craft a litigation plan that could help ease your fears while still operating in California Family Law's global framework.
If you cannot afford an attorney, do not feel comfortable retaining an attorney, or simply wish to do this yourself, the California Superior Court has great self-help resources online, in the courthouse, and through referrals to other agencies. Visit them at http://www.courtinfo.ca.gov/selfhelp/. You can also call their self-help number and they should point you in the right direction.
If you are looking for legal advice, you would have to contact a family law practitioner in your area as these matters are fact intensive and require much more than can be posted on a public website like Avvo.
Any individual seeking legal advice for their own situation should retain their own legal counsel as this response provides information that is general in nature and not specific to any person's unique situation. Circular 230 Disclaimer - Advice given in this response cannot be used to eliminate penalties with the IRS or any other governmental agency.
Sign up to receive a 5-part series of useful information and advice about child custody law.