There are a lot more facts to the situation here and so it is hard to answer. If the grandparents were awarded temporary custody, but that happened four years ago, it may be appropriate at this time to obtain a review of that order if there are changed circumstances. I do not recommend that you do anything until you consult with an attorney who reviews the temporary custody order. We can't do this over the computer, as much as I would like to help.
This answer is provided for informational purposes only. Actual legal advice can only be provided in an office consultation by an attorney licensed in your jurisdiction, with experience in the area of law in which your concern lies.
Your answer depends on what “temporary custody” means. Is there a court order giving them custody of the kids? If so, you’ll need to modify the order before you do anything. Simply taking the kids will get you into seriously hot water – contempt and kidnapping. If there is some other arrangement, the answer would change.
Take all of your paperwork to consult with an attorney.
Best of luck to you.
If you found this answer helpful, let me know by clicking the "thumbs-up" button at the bottom of this answer. It’s easy and appreciated.
This response is intended to be a general statement of law, should not be relied upon as legal advice, does not create an attorney/client relationship and does not create a right to continuing email exchanges.