If you have been granted guardianship by the court, then that was the appropriate time for the other grandparents to contest your appointment. They can, of course, file petitions that you aren't fulfilling your responsibilities, but if they're false, they're false.
Of course, you particular scenario cannot fully be analyzed or answered without a lot more facts, including: Was a Preneed Declaration made by the parents? Under what circumstances did your grandchildren become your wards?
While the other grandparents can submit a challenge that you are not adequately fulfilling your duties (fiduciary, etc.) as guardians, they'll have to petition the court to have alternative guardians appointed. While there is no way to protect them from doing this, by fulfilling your duties and possibly being "transparent" to ensure the other grandparents that all is well, you should not be bothered.
Obviously, if you have been appointed guardian, then you have an attorney, and further questions can be asked of her/him.