I don't know what you signed to "hold the car." But if you did not sign the retail installment agreement I don't see how they can hold you to the purchase.
I agree with my colleague, the answer depends on exactly what you signed, and you will at the very least be obligated to pay something for having this car "held" so it couldn't be sold to someone else.
Since you obviously continued shopping and didn't consider yourself bound by the 1st deal, it's way past time for you review what you signed, preferably with a consumer lawyer's help.
Avvo doesn't pay us for these responses, and I'm not your lawyer just because I answer this question or respond to any follow-up comments. If you want to hire me, please contact me. Otherwise, please don't expect a further response. We need an actual written agreement to form an attorney-client relationship. I'm only licensed in CA and you shouldn't rely on this answer, since each state has different laws, each situation is fact specific, and it's impossible to evaluate a legal problem without a comprehensive consultation and review of all the facts and documents at issue.
Often, those deals are contingent on financing. Chances are you signed something that said there was no deal if financing was not secured. As a contract matter, it is one thing to hold an offer open and it is quite another to accept the offer. I think an attorney might be able help you figure out what's going on here. But, I suspect that if you make it clear to the dealer you don't want the car and plan to notify any prospective lender as much, the dealership will rather quickly realize it is unwise to pursue things further. (This statement does NOT create an attorney-client relationship. Only an express agreement between you and the attorney can create such an agreement).