Is it possible for one sibling to cancel, change, or "take over" an irrevocable nongrantor trust w/o consent of the others?

Your brother may file a court petition to have the current trustee removed and have another trustee including himself appointed. The trustee and the beneficiaries are entitled to notice and an opportunity to be heard before the change can be made. If you are a beneficiary of the trust, you may serve and file an objection to the petition, and appear at the hearing to oppose it. You should consult with an attorney to discuss the particulars of your matter and for any specific legal advice.

First, the document is irrevocable, meaning that no one can change it. Second, who is the trustee and are they acting without due care and diligence. At best, he could attempt to remove the trustee for cause or malfeasance, but based on what you state there does not seem to be any bad behavior. Additionally, generally, these are tough cases to prove.
He is not in a position of strength. However, you should immediately hire an experienced estate planning/litigation attorney to discuss this matter in more detail and review any "papers" you get from him. From what you have stated, you are still in the drivers seat, but fight fire with fire-if he has an attorney, well, you get yourself an attorney, and by all means get a better one than he has!

LEGAL DISCLAIMER
Mr. Fromm is licensed to practice law in PA. The response herein is not legal advice and does not create an attorney/ client relationship. The response is only in the form of legal education and is intended to only provide general information about the matter within the question. Oftentimes the question does not include significant and important facts and timelines that if known could significantly change the reply or make such reply unsuitable. Mr. Fromm strongly advises the questioner to confer with an attorney in their state in order to ensure proper advice is received.
By using this site you understand and agree that there is no attorney client relationship or confidentiality between you and the attorney responding. This site should not be used as a substitute for competent legal advice from a licensed attorney that practices in the subject area in your jurisdiction and with whom you have an attorney client relationship. The law changes frequently and varies from jurisdiction to jurisdiction. The information and materials provided are general in nature, and may not apply to a specific factual or legal circumstance described in the question or omitted from the question.
Circular 230 Disclaimer - Any information in this comment may not be used to eliminate or reduce penalties by the IRS or any other governmental agency.

I agree with the other attorneys. Unless the trustees are not doing their jobs, are self-dealing, or breaching some other duty, I don't see what his problem is.

Any individual seeking legal advice for their own situation should retain their own legal counsel as this response provides information that is general in nature and not specific to any person's unique situation. Circular 230 Disclaimer - Advice given in this response cannot be used to eliminate penalties with the IRS or any other governmental agency.