The old saying goes that anybody with a pen and a filing fee can sue anyone. Translation: He can sue you for whatever he wants, but whether he can prove it is another story.
If he doesn't have anything in writing stating that it was a loan or that you understood you would be paying it back, then it will be his word against yours and very difficult to prove it wasn't a gift that simply flowed from being in a relationship together and helping you meet a need that existed at that time. You can't get a gift back.
You certainly could reach some compromise to avoid the hassle of court, but ultimately any threat he makes about legal action depends on his ability to carry the burden of proof that this was a loan. IF you actually get served with legal papers, talk to an attorney.