Property questions are quite fact sensitive. There is a concept called "adverse possession" which states that if you use property that is not yours, and the use is "open, notorious and adverse" after twenty years (in developed areas) the property becomes yours. The fact that you knew you could not plant shrubs or put up a fence, however, might indicate that you knew the property was not yours, and that you acted accordingly. Under the circumstances you describe, without knowing more, i don't think the property transferred to your ownership; but any attorney would need to know more facts before giving you a definite answer.
If you think this post was helpful, please check the "good answer" button below! NOTE: This answer is made is for advisory and/or educational purposes only. By using or participating in this site, or posting a question and obtaining an answer, you understand that no attorney-client relationship is being established between you and the answering attorney, and there is no attorney-client privilege between you and the attorney. You should consult with a licensed professional attorney in your state. The law changes frequently and varies from state to state. The information provided in this answer is designed to be general in nature and is based on the facts stated in your question, and might change based on further information.
Adverse possession claims involving fence lines can be complicated - and the case law goes both ways. There is a line of cases in NJ which provides that fences are rarely located exactly on the property line and that a fence that is slightly off the property line may not always give rise to a claim for adverse possession. See Mannillo v. Gorski, supra, 54 N.J. at 382, 388, 255 A.2d 258 (involving a fifteen inch encroachment) ("[W]hen the encroachment... is of a small area and the fact of an intrusion is not clearly and self-evidently apparent to the naked eye ... [the] presumption [of adverse occupancy] is fallacious and unjustified."); See also Maggio v. Pruzansky, supra, 222 N.J.Super. at 573, 537 A.2d 756 (involving an intruding strip slightly more than one foot in width).
If this answer was helpful, please mark it as helpful or as a best answer. This answer is for general education purposes only. It neither creates an attorney-client relationship nor provides legal guidance or advice. The answer is based on the limited information provided and the answer might be different had additional information been provided. You should consult an attorney.