Tricky question. Certainly it would questionable if Judge to meet with opposing counsel without you or your attorney. It is implicit that your attorney is representing your interest so it him/her being there for the communication would not be ex-parte. There are alot of hearings or the like that do not require attendance from the party, however Judicial Rules encourage, if not require, that parties of interest be noticed of any substantive dealings involving your case, but this rule may not be applicable if there are non-substantive matters such as scheduling, procedure, etc. being discussed. Just ask or write to your counsel and tell him your attendance is required for all future matters where Judge is involved, either telephonically or Live, and it should not happen again.
Your issue really involves communication between your lawyer and yourself. All the lawyers in a case can have a conference with the judge without their clients being present or even knowing about it. This is common in all types of litigation. With all due respect to Mr. Tropp, I would have to disagree with him. If you insist on being present for every conversation your lawyer has with the judge, you will greatly inconvience everyone involved, including yourself, and make things much harder for your lawyer to handle your case. If you don't trust your lawyer then get another one.
This answer is intended as general information and not as specific legal advice. If you want to have a free consultation with me, please contact me through AVVO.