Did you memorialize the agreement on paper? It sounds like he is working in exchange for rent. To terminate the tenancy, you would have to give him 60 days notice since he was there over a year. If he is causing substantial damage, you can serve him with a 3 day notice to perform covenant or quit. In other words, stop the vandalism. The burden is on you to prove this, and so it might be a better strategy to just serve the 60 day notice. You generally don't need a reason to terminate the tenancy. You can find more here... http://www.sandiegoevictioncenter.com/Unlawful_Detainer_Process.html
Scott Rights, Esq.
CA is not my state, but in my state, not only can you evict tenants, but you can also evict persons who once had a right to legally be at a property but now no longer do.
In order to terminate this right in CT, I'd do the following (a)tell him that you no longer require him and want him to leave and see what he says; (b) send a certified letter to him memorializing your demand (nicely but in no uncertain terms) that he leaves; and (c) if he does not leave serve him as you would a tenant, with eviction notices and process.
Your rules may be different in CA, but the basics of my advice will probably be sound.
The proper procedure would be to use the unlawful detainer laws of California to evict. This will require an attorney because your situation is more tricky. Moreover, as Attorney Rights indicates, you might have to give a 60 day notice of termination.
However, if you can convince the caretaker to move out earlier, you will not have to incur the time and expense to evict.
The information presented here is general in nature and is not intended, nor should be construed, as legal advice. This posting does not create any attorney-client relationship with the author (who is only admitted to practice law in the State of California). For specific advice about your particular situation, consult your own attorney.