The answers to these questions depends, in part, on what the trust itself says. Usually the trust provides that as long as the primary beneficiary is alive, you only need to account to him. Likewise, he is usually the only person who has the power to remove and replace the trustee. BUT all trusts are written differently. It's possible that your father's trust states that the "contingent" beneficiaries have these powers if your father is deemed to be incompetent.
If your father is competent, I urge you to protect yourself by providing him with accountings and getting him to sign receipts showing that he received them (I would have him initial each page of the accounting so that there's no doubt down the road).
"Bad blood" between siblings is one of the reasons that I strongly suggest that my clients use a professional trustee - it costs a few dollars more on the "front end", but it saves thousands of dollars and untold grief on the "back end".
The information you obtain at this site is not, nor is it intended to be, legal advice. You should consult an attorney for individual advice regarding your own situation.