You may be responsible. As a landlord you have a duty to maintain the premises in a habitable condition. Vermin are definitely a breach of that duty. The best bet is to maintain your property o avoid any issues with your tenant. If you are able, depending on your lease and your county, do not renew the lease to avoid this tenant. You should consul with a landlord attorney in your county to provide further information.
NOTE: This answer is made available by an attorney licensed to practice in the state of California. The communication is intended for educational purposes only. By using or participating in this site you understand that there is no attorney client privilege between you and the attorney responding.
However, if the tenant is contributing to the problem by being unclean, leaving food out, etc., then he is breaching his duties under the lease (almost certainly). You have a duty to evict him if those breaches are continuing, as you do have the liability for an uninhabitable unit. However, it sounds like you think he's just a nag and you dont like responding to nonsense complaints. If he's a nuisance who makes meritless complaints I think you could get him out on those grounds as well.
On the hand if he's not a contributor and the building is just infested, you definitely should plan to foot the bill.