The court order is the default parenting time arrangement between the two of you. If the two of you agree to something different- more or less parenting time for Dad, etc. that's fine. However, when you two can't get along, the court's order controls.
If he wants more time, it's up to him to do something about it i.e., convince you or bring a motion before the court for more parenting time.
The above is not intended as legal advice for your specific situation or intended to create an attorney-client relationship. Talk to a lawyer for advice tailored to your specific needs.
Sign up to receive a 3-part series of useful information and advice about child custody law.