As the registered owner of the vehicle, you have legal responsibility under California law to maintain liability coverage on your car, even if you do not personally drive it. Your insurance would be the primary carrier when you have a permissive user. However, if you did not have coverage, then I would expect your permissive driver's insurance carrier to cover for the cost... except that if he is a regular, or only, driver of the car and he did not notify his carrier that he was the exclusive driver of the car, his carrier may claim he was in violation of the terms of his policy and refuse to cover. If so, was this done to avoid paying a higher premium? If that happens, you may find yourself legally responsible and without any insurance to cover for the accident. How did you end up with this arrangement with you owning a car but someone else as the exclusive driver?
As I am licensed in Florida and Vermont, I can only provide you with some general guidance. California may well be a compulsory insurance state. Depending upon how frequently or exclusively this person drove your vehicle, and what he told his insurance company about the arrangement, he may or may not have insurance coverage. If he misled his insurance carrier, they may well void his coverage.
I suggest that you may wish to consult with a civil defense attorney in your area who is familiar with California insurance law before you sign any affidavits or say anything further.
Sign up to receive a 3-part series of useful information and advice about personal injury law.