Do I need to pay a contractor who did unauthorized work?

I had a plumber do an estimate on replacing a water heater and putting in bathroom vents and associated work to vent the fans to the outside of the house. After getting the estimate I told them to go ahead and do the water heater only.

I then went and got another estimate for the work from a company that was 1/3 the cost. This company is a trusted, respected company.

The first company then went and did the water and then did the vents even though I said water heater only. I never signed anything with them authorizing the work but did give verbal go ahead on the water heater as I stated.

I don't want to pay the contractor for the vent work. Do I have legal grounds to refuse payment?