(1) Land sales must be in writing. Oral contracts are generally invalid.
(2) Adverse possession requires 21 years of open, notorious possession of the land in question.
(3) YOU NEED A LAWYER NOW, because this is over your head.
It sounds like you live somewhere above the Back Road where there are a lot of properties where the owners maintain some portion of another's land. I hope you had a title insurance company insure your purchase (although it does not sound like it) - that might help with the upcoming legal fight. To answer the question, the neighbor certainly can make a claim, and might have a VALID claim, but that will be up to the Court to decide. You should contact your attorney immediately to review all of the history of the properties in question and help you decide on a strategy.