I just purchased a business and the broker spent the ernest funds and is suing for closing fees. We did not use him for dishones

Both the seller and buyer had 2 different contracts. The ernest funds were spent by the broker before closing. When we were out of contract and we verballly negoiating the contract broker said we were out and the ernest funds were his found out at closing. He then wanted me to come up with the funds he spent saying I owed him his 10%. this was part of the verbal agreement when we were out of contract that buyer would pay the broker fees. We did not close with him as he did not let me know he spent the funds showed he sent an email which no one saw until closing. The seller still wanted us to purchase the business and we closed without the broker now he says I owe him more funds. So why is that contract still in agreement and my when I was in verbal agreement was no contract. Also both the seller and broker had 2 different contracts which none of us signed the others contract. Broker said the very first paper I signed saying he will show me a business means I had to buy through him. Does this means he was able to do what he liked. He negoiated not in my favor. And did not send a letter that he spent all the funds from the ernest money I put down.