I think you should contact an attorney and attempt to get it all back; even if you gave her cash. If you gave it to her and she denies it, she would be perjuring herself. Call an attorney, paying an attorney is better than just losing the $4500.00 by giving up.
the memo that it was a loan is helpful. I agree that you should pursue the whole amount. It may be a tough case without loan documents or loan terms etc. It is a matter of credibility and who the court will believe. It is your burden of proof though. Even if you win you now need to find something to collect from. Something tells me she is not going to right a check to you as she leaves the courtroom. A judgment is only as good as its collectability.
You had best hire a lawyer and see where it goes. As a matter of evidence she would have to trace where she got the down payment. As to whether it is gift or loan: judges and juries can often discern the truth when you have conflicting testimony. Having written "loan" on the check bolsters your position to the second payment. Now find a woman who likes you for yourself and not your $11,000.
If you can keep it under $10,000, you can do a simple small claims filing. Otherwise, you would need to file as a regular claim in civil court. It is typically preferred to retain an Attorney to assist in the filing and in representation of your claim(s). I do agree that what you wrote in the note will be strong circumstantial evidence, though it will not be as convincing as a contract. As to the down payment on the house, that will be difficult to prove.
Your ex girlfriend could probably file bankruptcy and discharge your claim quite easily for $1500 or less in legal fees and court costs, even if you won on the issue of whether these money transfers were loans rather than gifts. While you would likely win on the checks representing loans (due to writing "loan" on the memo), you therefore might still wish to cut your losses by writing this off to experience. As for the undocumented cash transfers to her, litigating the loan issue in the context of any relationship wherein large gifts would seem appropriate would be messy and uncertain if she fought your claim. That is why you should also prepare proper documentation any time that you are making large loans to anyone, and consider also obtaining collateral pledges (in writing and perfected by recording with the register of deeds) in order to guarantee your loans. Please do not assume that I am your attorney because of my response here. Call my office in Racine (262-633-3090 or email email@example.com) for clarifications, but short of such additional arrangements, I will not be taking any action on your case. See me on the web at www.jayknixonlaw.com, or view over fifteen years of my past answers at http://www.lawguru.com/answers/search/attorney/jknixon, as well as past AVVO answers at http://www.avvo.com/attorneys/53401-wi-jay-nixon-1529181/answers.html. Answers may contain attorney advertising materials.
My answer here does mean that I am representing you, so be sure to consult your own attorney before deciding on what you should do, or contact my office at 262-633-3090, 333 Main St, Racine, WI 53403, during business hours if you desire formal representation. See me on the web at www.jayknixonlaw.com, or view over 15 years of my previous answers at http://www.lawguru.com/answers/atty_profile/view_attorney_profile/jknixonAttorney answers may contain advertising materials.