Not enough facts to provide a definitive response. I think you're going to have to see a lawyer to go over the contractual relationship with the contractor and the details of the specific facts involved before you can get any real guidance.
However, assuming you have a valid subcontract with the contractor (and that the subcontract does not have any language prohibiting you from dealing directly with the client and setting penalties, etc. for doing so), you should be able to collect from the contractor for other work that was already completed and for which the contractor previously had agreed to pay you. It sounds like this contractor is not going to do so willingly, which means you may have to sue him to collect. If you file suit, it's certainly possible that the contractor may file a counterclaim against you -- whether valid or not -- that (if successful) might reduce, eliminate, or overwhelm the amount of damages you seek.
Best to get a lawyer on board.
DISCLAIMER: The information provided in this post is general information only and is not intended to be, nor is it, specific legal advice for the facts cited or for any similar facts. The reader is encouraged to seek the assistance of a lawyer to assess the complete set of facts and provide legal advise thereon. This post does not establish any attorney/client relationship, either with the person who submitted the actual question or any other person.
You need to talk to a lawyer, there is not enough information in your question. Don't wait! You may have some mechanic's or materialmen's lien rights that you could lose if you wait too long.
Disclaimer: This post does not create an attorney-client relationship. This post is for informational purposes only and is not intended as legal advice. This post is not a substitute for holding an in-person initial consult with an attorney.