You have a couple of options.
In TN there is a six year statute of limitations on a debt. The time starts with the date of the debt if there have been no payments made. If you have made any payments, the clock restarts from the date of the payment. You mention the loan was around 6 years ago. If you have made no payments and a lawsuit is not filed until past the 6 year mark, you can raise the statute of limitations as a defense. Note if you were to make a payment now, even a small one, the clock resets to start over on the 6 years.
Another option may be to file bankruptcy. I this is your only debt, a few thousand, this might not make sense but if you have other debt you are struggling to pay, it might be an option to consider.
You might consider making an appointment with an attorney who practices in consumer debt and bankruptcy to discuss and review. Many will do an initial meeting at no cost.
This answer does not constitute legal advice nor form an attorney client relationship. I am not your lawyer. If you have a legal issue in Tennessee you may contact me for a free consult.
Of course an individual that lent you money has the right to sue if you don't repay the debt. However, if the person does sue you, collecting from you may be a problem. State laws called exemptions protect certain items from being taken from you in order to pay off a court judgement & I am posting a link to a general description of these items below. Hope this perspective helps!
Sounds like it is simply a "threat" and nothing more. He said-she said type of situation with nothing in writing. It would be very difficult to prove that you owe him anything.
I am not your attorney nor is any answer I may provide legal advice. You may contact me directly for legal advice only if you are a resident of Tennessee insofar as I only practice in Tennessee.