No, not whenever he feels like it. He probably is allowed to come into the property as the landlord's agent to make reasonable and necessary repairs. A landlord is allowed to come onto the property, upon giving reasonable notice, to make repairs to keep the place habitable.
A landlord could probably rely on that right to make occasional (I'm talking maybe quarterly at best) inspection of the premises (checking/replacing air filters, replacing fire extinguishers, etc.).
I guess it depends on how often he's coming in and what exactly is his purpose for coming in. If this is happening more regularly than once every three months, I think you need to express, in writing, to your landlord that her husband's frequent visits to the residence is a little intrusive and that you'd prefer he give more notice and perhaps wait so that multiple issues can be addressed in one visit as opposed to every time he thinks of something he wants to check on.
I think you may want to consult with a lawyer about your situation and see if it gives you grounds to terminate your lease. It sounds to me like you may.