As Mr. Spirtos observes, you have not provided enough information to properly answer this question. You also have some misunderstanding about how the law applies. An "owner builder," under California law, is a person who owns the property and obtains a permit to do construction work in his or her own name. The owner can hire people to do the work for him or her, but he or she is the permit holder on the job. If there is a building contract permit on file in your case, it was pulled by the contractor in his name. Yes, this would mean, by implication, that the contractor is claiming he has a contract with you (and he would have to be a licensed contractor to be able to obtain such a permit).
You may want to look at the California Contractors State Board website: http://www.cslb.ca.gov/Consumers/KnowRisksOfOwnerBuilder/
It is hard to understand what the actual question or dispute is here, but the contractor does have mechanics lien rights if he is the permit holder and has done work on your property which he claims is not paid for. You don't say that a lien has been filed or posted, and there are time limits that apply.
You do say that a contractor did work on your property, and it seems clear that you had some kind of agreement with the contractor (perhaps informal, but something that caused him to go forward and work on the property). If the contractor was not licensed, he can't pull a permit in his own name. An unlicensed construction worker can do work for someone, without a license, on a "time and Materials" basis but not for a fixed rate contract.
Unless the work is highly defective or the dollar amounts are very considerable, you should try to sit down with the contractor and try to work out your dispute informally. You would be well served by having a consultation with an attorney experienced in construction law in your area before trying to discuss it informally, so you know what the alternatives and the cost of those alternatives will be if you don't work it out informally. Be sure to bring all paperwork to the attorney when you have the consultation including the contract, if any, the permit, and anything in writing that you have pertaining to the project or the dispute. Otherwise, the attorney can't be sure what the situation really is. Good luck.
There is not enough info to give you a determination on your specific issue.
I guess the contractor can claim anything he wants. If he is claiming a contract with you, then he will need some proof that there is a contract, whether oral or written. A contractor is required to have a written agreement if the work is for home improvement. Not having one might not mean that he could not still sue you. If this is a commercial property, the rules are not as strict.
I don't whether your building department requires a verification or declaration from the owner to pull a permit. Contractors pull permits all the time for owners.
Take everything to a local attorney for a short consultation.