Depends on what was stated in writing, or if nothing, then what you could prove was orally agreed. You should consult an attorney that does construction law nearby. Keep in mind that contractors can take out liens on property for lack of payment.
First, read your contract. Your contract with the builder should adopt specific plans and specifications -- basically the design for the home and what you are getting for your money. If the buider has changed the design without your approval, the builder may have breached the contract. Your contract should list your remedies in case the builder defaults. I would have to review the contract to determine what the contract allows. I recommend that you write to the builder by certified mail, and complain about the changes. You probably should approach the designer of the home or an architect or another builder that you know to confirm that the changes are significant. If the changes are significant enough, you may consider writing to the builder and advising that unless the builder remedies its defaults, and restores the home to what you had agreed to buy, you will declare the builder in material breach of the contract and seek your remedies under the contract. One remedy may be the refund of your down payment and termination of the contract.
By the way, if you intended to live in the home being constructed, it could be considered your homestead. Unless the builder complied with the homestead mechanic's lien requirements of Chapter 53 of the Texas Property Code, the builder may not be able to file a mechanic's lien against the home. But, if you just want your money refunded, whether or not the builder can legally file a mechanic's lien is not relevant. For more on mechanic's liens, feel free to visit my web site: http://www.theconstructionreport.org
Bottom line, the changes have to be material, and contrary to what you agreed to in the contract, before you can legitimately complain about them.