To establish a claim for adverse possession, the claimant must establish that the neighboring property owner had actual notice of the adverse use throughout the statutory period or (2) that the claimant used the land in such a way that any reasonable person would have thought that he owned it. Because you can establish adverse possession by showing that a reasonable person would have recognized your use of the disputed property (as opposed to actual notice), the fact that the lot was vacant is not a bar to your claims. Further, a lot line adjustment is not necessary to establish adverse possession.
I'm afraid you probably do need a real estate attorney -- although it will not be me, because I am not licensed to practice only in Maine, and not in Washington state.
I briefly surveyed the case law in Washington to see if it was consistent with Maine law. I found that it was virtually identical, except that the period of occupancy needed to establish title by adverse possession is only 10 years in Washington, as opposed to 20 years in Maine.
There is no such thing as a "clear-cut" adverse possession case. If the abutter's survey is correct -- which it very well may not be -- then you have the burden of proving that your use of the land up to the fence line was hostile, open and notorious, actual and uninterrupted, and exclusive for the entire ten-year period. Whoever has the burden of proof has something of an uphill battle.
With that precaution, I can tell you that your neighbor's arguments are incorrect. It does not matter whether anyone was actually living on your neighbor's land during the ten-year period. It would have been helpful if the neighbor had had actual notice of your use of the land up to the fence, that is not necessary. It is sufficient if you used the land in such a manner that any reasonable person would have thought you owned it. The fact that the owner left the land vacant does not free him or her of the obligation to keep an eye on the property. If a hypothetical "reasonable person" who saw your use of the land would have thought that you owned up to the fence, then the use is sufficiently open an notorious to allow title by adverse possession to ripen.
Neither does it matter that you or a predecessor in title might originally have made a mistake in putting the fence where it is. So long as you and they treated the land as your own, and all the other elements of adverse possession are present, you still qualify for title by adverse possession. Under Washington law, placing a permanent object such as fence on someone else's land is considered extremely strong evidence that you are claiming it as your own.
It makes no difference whether you have had a "lot line adjustment." (I'm not exactly sure what that is, but I assume it is some sort of proceeding to render a new boundary line "official" or to put in on the public record.) If the elements of adverse possession persist for 10 years, title by adverse possession passes by force of law, with no need for any formalities whatsoever.
I strongly recommend that, before assuming that you have to deal with an adverse possession case, that you have your property surveyed. Most people seem to think that whatever a surveyor says must be true, due to the apparently scientific nature of boundary surveys. That is not true at all. A survey is just one surveyor's opinion about where the boundary line is located. An equally-qualified surveyor might find that the boundary is in a significantly different location. Surveying is as much an art as a science, and calls for considerable subjective judgment on the surveyor's part. If you have your property surveyed, you may discover that the fence is actually on the boundary line, just as you thought it to be.
By answering this question, I have NOT formed an attorney-client relationship with you or with anyone else. My answer to your question is only a general one, and neither you nor anyone else can rely upon it as legal advice that would be appropriate for your particular situation.
Assuming that the survey is correct (they are not always so) it appears that you have a strong case here. The other answers are correct in what they say.
It is also worth noting that in an action to quiet title to the disputed property, the prevailing party now may recover attorney fees and costs. i.e. if you perform a survey and it appears that you have been using/maintaining the property in question, you could remind your neighbor that if you prevail in an adverse possession suit, he will likely have to pay your attorney fees and the cost (survey, etc) of the suit. The statute is relatively new. It is RCW 7.28.083. Granted if you do not prevail in the suit, you may be responsible for his fees under the same statute. My point being, that if your case is as strong as it appears, then it might make sense to alert the neighbor that you will not back down and see what he says. He may ultimately take the steps to have the land titled in your name.
Posting questions and reviewing information received from AVVO does not create a lawyer client relationship and such information should not be construed as creating such a relationship.
A survey identifies the boundaries of property as legally described in a deed. It is does not, however, reflect the rights and interests of parties in possession as may be determined by a court. Based on the facts you described, it sounds as though you may have a claim to quiet title by adverse possession. In order to quiet title by adverse possession you must prove several elements by a preponderance of the evidence, i.e. that the adverse possession was open and notorious, exclusive, hostile, actual, and continuous for the prescriptive period of 10 years. The elements tend to blend together with the test being whether the claimant used, possessed, improved the property in dispute as a true owner would have given the nature, characteristics of the property. Your neighbor's comments demonstrate a lack of understanding of adverse possession. You may want to meet with an attorney to prepare a letter to the neighbor in order to educate the neighbor regarding your interest in the property and the potential consequences of interfering with that interest. I hope this was helpful and good luck.