Withholding payments until the matter is "resolved" is not to be undertaken without first understanding your contract and your rights under the contract, the status of construction, your rights to retainage, and often requires proper notice. The affidavit may provide some ability to make claims but it does not assure that your home will be without liens or that subcontractors do not get paid and, absent some specific circumstances, it does not provide you the protection you truly need. Depending upon the scope of construction, if your contractor is not state licensed and is required to have a state license you may not be obligated to pay anyone. If your contractor is state licensed you may rights to file complaints with the Board in addition to other rights and remedies. My strong suggestion is that you consult a construction attorney who knows the ins and outs of residential construction for specific advice and a strategy to cover your particular situation.
Disclosure: This answer and any information contained in this answer is not intended to be treated as legal advice. It is for informational purposes to educate about legal issues. You should contact an attorney for specific legal advice for your situation. This posting does not create an attorney-client relationship or privilege of any kind. This attorney actively licensed only in the State of Georgia.