If you have no written agreement with this particular real estate agent, and you have not made an offer to purchase the same property with this agent, then you are free to change real estate agents at any time. However, this (bad) agent still represents you in connection with the house that he did show you.
The information presented here is general in nature and is not intended, nor should be construed, as legal advice. This posting does not create any attorney-client relationship with the author (who is only admitted to practice law in the State of California). For specific advice about your particular situation, consult your own attorney.