He can sue you but what are his damages? You offered to pay for changing the locks. If you do that, what else can he want?
No damages means no case. If he does sue you, he'll have to prove that he was somehow harmed, and it's hard to imagine that the amount will be greater than the cost to change the locks.
Sounds like a misguided bully to me.