Have you got a written contract? If so it would be with the contractor, not the subcontractor, and you're right, any dispute between them is not your responsibility. Pay the contractor only what you agreed to pay for the job, less any payments you've already made, but don't do that without getting an unconditional release and waiver of lien from your contractor which states that all subcontractors and vendors have been paid in full.
Disclaimer: Please note that this answer does not constitute legal advice, and should not be relied on, since each state has different laws, each situation is fact specific, and it is impossible to evaluate a legal problem without a comprehensive consultation and review of all the facts and documents at issue. This answer does not create an attorney-client relationship.