It depends on what the contract says. You first look at the start date of the work to be performed. If the contract has the start date defined in the agreement, it is possible that you could terminate the agreement for breaching that provision. Secondly, look at the requirements for terminating the agreement. You may have to give a notice to the contractor of his breach and give him a certain period of time stated in the agreement to cure this breach before you may terminate the contract. At any rate, you will want to give the contractor written notice of the termination of the contract and state the reasons why.
The contract controls. You may want to check the home improvement contractors record with the Better Business Bureau, and also make sure that he has a license with your particular town. If he is not licensed by your town, and such a license is required, you can cancel the contract with impugnity. Note, not every Long Island Town has a requirement that the home improvement contractor be licensed by the Town. If your town does not have one, then you should check with the County to see that he is licensed.