Does this trust already exist or is it in the event of your death? If it is your estate plan you can choose who the trustee is. It may be a good idea to coordinate with your ex on these decisions if you can.
If the trust already exists, it may be possible to petition a court to change trustees or look to the trust document for guidance as to when trustees can be changed.
This is not legal advice nor intended to create an attorney-client relationship. The information provided here is informational in nature only. This attorney may not be licensed in the jurisdiction which you have a question about so the answer could be only general in nature. Visit Steve Zelinger's website: http://www.stevenzelinger.com/
The terms of the trust will determine if a co-trustee can be removed and replaced. The trust agreement should address where both co-trustees must act jointly or whether either may act on behalf of the trust. You need to take the trust instrument to an experienced trust and estate atorney who can advise you.