Yes he can sue you in NJ since you live in NJ. Doesn't really matter where he lives or where the agreement was made. You don't have to prove anything. As the Plaintiff, it is his job to prove whatever he alleges. If there is no written agreement or other substantial proof, I doubt he will be able to convince the court that you owe money
This answer should not be construed as creating an attorney-client relationship, and is for informational purposes only, not legal advice.
When you consult with an attorney, you may want to explore whether the applicable statute of limitations will make short work of the lawsuit.
Disclaimer: Nothing stated herein is legal advice. For legal advice, consult an attorney; I am not your attorney at this time. This response may constitute ATTORNEY ADVERTISING which has not been approved by the Supreme Courts of New Jersey or Pennsylvania or the Court of Appeals of Maryland. I am a federally-designated “debt relief agency” that provides, where appropriate, relief under the U.S. Bankruptcy Code.