Most legal issues surrounding your situation will be governed by the contract itself and the laws of contract in Georgia. Without studying the contract, it is not possible to answer you questions unconditionally. However, as a practical matter, you should demand, in writing, a copy of proof for the damages that the general contractor is claiming. Some questions that come to mind are: Is the GC seeking the full value to him under the contract (as if it was fully completed)? Does the contract have a liquidated damages clause? Why did you terminate the contract? What does the contract say in regards to what amounts of monies are owed if the contract is terminated prior to the completion of the house? Was there a builders' draw in the contact? Was it followed?
Finally, if you fired the General Contractor and terminated the contract for good cause, it is not likely that you would owe any more money than what you paid under the draw schedule. In order to properly protect your legal rights, you should retain an experienced Georgia lawyer to review the contract, and determine who owes whom what. If the General Contractor was negligent, it is not likely that you would owe any more money and you would possible be entitled to recoup some of the money you already paid him. But, once again, so much depends on the terms of the contract and the specific facts of your case. Good luck.