She cannot legally get your parenting time with your son without a valid court order or your agreement. If there is something in particular that concerns you, this is best discussed with a family law attorney. If she alleged that your environment was endangering your son, she could keep him from you for up to 14 days pending a hearing. [See C.R.S. 14-10-129(4)].
Mr. Littman is right on point. Why do you fear that your wife is going to get your parenting time when you have a joint agreement to the contrary?
The information provided in this answer does not create an attorney-client relationship and is not considered to be legal advice. Mr. Leroi answers questions on Avvo because he strongly believes in public service from his years as a judge, magistrate, and prosecutor. If you need to ask any follow up questions because my answer did not fully address your question, feel free to call Chris or post an additional question. Thank you.