Is there a real estate agent? The agent usually accompanies buyers when they visit the property. I don't think you are going to get anywhere with such a claim. If you want the deal, you will probably kill it. If you don't want the deal, you can probably terminate under the agreement's terms. Talk to your agent. If no agent, talk to a lawyer. The agreement will govern whether you are entitled to terminate.
Clifford L. Tuttle, Jr.
Attorney at Law
The foregoing answer does not constitute legal advice and does not create a lawyer-client relationship. Answers are based upon the facts stated and may change if there are additional facts not contained in the question.
If the inspector went into your house without any permission, then that's a trespass. I don't know what you would want to do about it at this point, other than telling your real estate agent and having the agent make it very clear that this should not happen again. If you pursued it in court, (which I don't recommend) you should be entitled to damages, which would be nominal (unless he broke something).