Reasonable or not, you should stay away from this person (i.e., stop taking walks in their neighborhood, at least by yourself). If you have reason to believe that you might be in danger, then, yes, report it to the police (although I'm not sure what they would do, really).
Ms. Berjis is licensed to practice law in the State of California. The laws of your jurisdiction may differ and thus this answer is for informational and educational purposes only and is not to be considered as legal advice. Since all facts are not addressed in the question, this answer could change depending on other significant and important facts. This answer in no way constitutes an attorney-client relationship.
I recall you posting a similar post a few weeks ago--except in that post I recall you were the individual watching the person in red shorts and a T-shirt.
None of the guidance you get here will be any different than with your first narrative.
NOT LEGAL ADVICE. FOR EDUCATION AND INFORMATION ONLY. Mr. Rafter is licensed to practice in the Commonwealth of Virginia and the US Federal Courts in Virginia. There is no implied or actual attorney-client relationship arising from this education exchange. You should speak with an attorney licensed in your state, to whom you have provided all the facts before you take steps that may impact your legal rights. Mr. Rafter is under no obligation to answer subsequent emails or phone calls related to this matter.