If the contract provides that you are entitled to recover the builder's deposit if you cannot obtain financing, the builder should obviously be refunding your money. You should contact a local civil litigation attorney. A letter may be enough and, if not, he or she may be able to help you with filing suit. Feel free to call if you have any questions.
T. Shawn Howard
The information provided should not be considered legal advice and does not create an attorney client relationship. I am not licensed to practice in any State other than North Carolina. The results of your case will depend on the presentation of evidence, the law and other factors that may change depending on an in depth analysis of the facts of the case. Please see an attorney before making legal decisions.
Your question is difficult to answer without seeing the written contract between you and your builder - assuming you have one. You may have bigger problems than just forfeiture of the deposit. If the builder has a claim for breach of contract against you, it may seek to recover its net profit from this project, plus any available contractual damages. See a construction attorney in your state as soon as possible.
This response is intended to provide general legal information rather than specific legal advice, and does not create an attorney-client relationship.