If your software did not work, and you knew or should have known that your software would not work, you will have a hard time securing payment.
You really need to retain an attorney to evaluate your contract, and the pertinent paperwork and communications. There could be enough of your software and intellectual property that is useful, that you should be paid for some or all of your services.
The key factor is the language of your contract. It *should* spell out how final payment upon termination should be handled. If the language is insufficient to address your situation then it can depend on a number of factors. I imagine the payment owed is more than a few hundred dollars. If so, it is probably worth your time and money to speak with a local business lawyer about your situation.