This is definitely not a question that can be answered via a written post.
California law in regards to boundary disputes is complex and technical. Please note, typically location of a fence does not control for boundary lines.
There are too many variables to discuss in a written post forum such as this. You should discuss this matter with legal counsel as the law in this realm is complex and oftentimes, the best result is achieved through negotiation and settlement and/or some other form of alternative dispute resolution.
I have posted some information regarding boundary disputes in my avvo legal guides for your review.
Hope this helps.
If this is really just about the removal of ground cover, it's a very heavy handed approach. If it's just ground cover and it's on their property, why don't they just remove it themselves? It would be a lot less expensive then hiring an attorney to sue you. In fact, if the amount of damages is less than $7500, then they are in small claims court and must represent themselves (unless they want injunctive relief but you said damages). So I sense that there is more to this story. In terms of you gaining ownership of it, in California, you can't get adverse possession over another's property unless you've been paying the property taxes on it. There are various other possibilities, but I concur with the prior person who answered this -- boundary line issues are complex and there is a lot of law on the subject. I'm also not clear on what you want. It sounds like it would be better for you if they maintained their own property, and it sounds like they now want to do that. If what you really want is a prescriptive easement over this part of the property, then it's a different story.