It is difficult in that nothing is in writing and in that this is family.
But, you are probably in a better legal position as you had a reasonable expectation that the agreement was you are renting the whole property.
Enforcing that right is tricky.
Another option is to give your grandfather written notice twenty days in advance that you are moving out at the end of a given month.
It is true that you need a written agreement when you enter a lease. This is always a good idea, as it gives you a basis upon which to stand to enforce your rights as a tenant.
In any case, in all residential landlord-tenant relationships in Washington, the landlord owes to the tenant the right of quiet enjoyment of the property. That is, the landlord cannot do anything that unreasonably interferes with your use and enjoyment the property, and must not allow third parties to impinge on your use and enjoyment of the property.
Assuming you can prove in court that you have an existing landlord-tenant relationship with your grandfather, you should be safe to write him a letter and tell him that your use and enjoyment of the property is being interfered with by the people in the backyard, and that you expect him to have them removed. If he fails to do so, you should have legal basis to enforce his duties to you in court.*
*This answer is not intended as legal advice but merely general legal information. This answer is not intended to create an attorney-client relationship, and no such relationship has been formed as a result of this communication.