It is not illegal, and a one time incident is not harassment. Politely tell him you do not want him to do it again. If he does, then have a lawyer send him a cease and desist letter. If he does it again, then you may have enough for some type of restraining order.
We do not have an attorney-client relationship. I am not your lawyer. The statements I have made do not constitute legal advice. Any statements I have made are based upon the very limited facts you have presented, and under the premise that you will consult with a local attorney. This is not an attempt to solicit business. This disclaimer is in addition to any disclaimers that this website has made. I am only licensed in California.