The burden of proof is on the person claiming the existence of the agreement to prove that it actually does exist and is contractually binding. If it's a verbal agreement, yes, there are ways to do that but it's a pain in the petoot. Because of this, I suspect he's blowing a lot of bad-smelling hot air at you, but you should probably retain an attorney just in case.
THE INFORMATION PRESENTED HERE IS GENERAL IN NATURE AND IS NOT INTENDED, NOR SHOULD IT BE CONSTRUED, AS LEGAL ADVICE. THIS POSTING DOES NOT CREATE ANY ATTORNEY-CLIENT RELATIONSHIP BETWEEN US. FOR SPECIFIC ADVICE ABOUT YOUR PARTICULAR SITUATION, CONSULT YOUR ATTORNEY.