IImder gemeral contract law, with an "unsigned" contract for home remodeling, the basic question is, whether there was a meeting of minds as to the essence of the agreement, that a reasonable person would believe that the essential terms of price and performance had been agreed upon. The fact that you both agreed upon changes that the owner had drew up, seems to indicate that there was an agreement. The fact that the contract was not formally "signed", may be countered by the fact that there were "pen and ink" changes. Also, the fact that he agreed to pay your gas expense to get to his propoerty, futhers your argument.
I would consult a local attorney in your jurisdicton, who can review your document and knows the local rules about home improvement contract requirements.
As an aside, you should always get a security deposit, that would cover your time not doing work on a job you can not do because of the work you schedule to do. In other word, your should set a deposit equal to or greater than your lost opportunity costs.