It would be unusual for an order from the judge not to specify exactly when the children is to be with whom. The order should specify which day and hours you have the children.
Since no one here has seen your orders, you need to review the orders and the other details with your attorney to find out your legal options.
If the grandparents have legal custody of the children, they likely can decide when someone else can see the children if there is nothing in the court orders specifying when someone else can see the children.
Do you have a copy of the Order that you reference? If you do, it should specify how much time right on the Order. If it doesn't, that's rather odd. How old is this Order? Did you attend the hearing where the Order was signed by the Court?
DISCLAIMER: This answer is provided for general educational purposes only. By using or participating in this site you agree and understand that there is no attorney client relationship between you and the attorney responding, and no attorney-client confidentiality. The law changes frequently, and varies from jurisdiction to jurisdiction. The information provided in this answer is general in nature and may not apply to the factual circumstances described in your question. The applicable law and the appropriate answer may be different in the State or States where the relevant facts occurred. For a definitive answer you should seek legal advice from an attorney who (1) is licensed to practice in the state which has jurisdiction; (2) has experience in the area of law you are asking about, and (3) has been retained as your attorney for representation or consultation. Your question and the attorney’s answer may be used for promotional or educational purposes.
Sign up to receive a 3-part series of useful information and advice about child custody law.