I agree with counsel. You get to choose what happens during your periods of possession.
I'd like to add that if you ex isn't being cooperative with you spending time with your child, you might need to speak with an attorney. You shouldn't be isolated from your child or bullied. You might need a lawyer to protect you.
This does not create an attorney-client relationship. You are free to follow or ignore my opinion as expressed on avvo.com.
If you have a custody order, you can enforce it. You do that first and foremost by reading it and knowing your rights and telling your ex "no" when she tries to infringe on them. Unless they are in the custody order grandparents don't have legal rights to possession of the child during your periods of possession. "No" is a complete sentence. Try it out on the ex. That said, if it is grandparent's 50th wedding anniversary...you can be flexible for special occasions.
This does not establish an attorney/client relationship
The grandparents do not have any rights unless they have an order giving them possession and access. From a legal perspective, they are simply people who are spending time with the children during your ex's period of possession, or that is what it should be. Recognize, you have an exclusive right of possession during your periods as set out in the order. If she denies your possession, you can and should bring a contempt action. Note also, it is a state jail felony for someone to deny possession to someone who has an ordered right of possession. The Dallas DA won't press those charges, other counties (Kaufman) will do it in a heart beat.
Protect your rights. Consult with a lawyer. One other thing, for a grandparent to actually get court ordered possession is a very very difficult expensive proposition for all involved.