You can file a police report, but many police departments don't like to get involve in these disputes and will tell you to pursue the matter in civil litigation. I have filed multiple lawsuits in this type of situation -- and I have won or settled 100% of the case; however, there is almost always a problem collecting the judgment.
These "fly by night" contractors usually have no assets, no home, no bank account in their name, and no regular paycheck. This means you might file a lawsuit and win a $15,000.00 judgment, but you will spend forever trying to find 50 cents to seize from the defendant.
Never hurts to consult with an attorney, but think very carefully before spending money on a lawsuit when you may not recover one cent even if you win.