You need to direct these questions to your attorney -- the one that handled the custody orders that resulted in the children being placed with Gram and entering a support order for you to pay gram.
If you don't have an attorney, you need one. Hire an attorney, immediately.
This question seems incomplete. Not only is the question incomplete the facts are as well. Either way, something does not make sense and I suggest speaking with an attorney. Judges do not typically just "take" children from a parent while that parent also has custody. That just doesn't legally make sense. Speak to an attorney in detail.
Communication with an attorney does not by itself create an attorney/client relationship or constitute provision or receipt of legal advice. Any communication with an attorney should be considered informational use only, and should not be relied upon or acted upon until a formal attorney/client relationship is established via a written agreement.
Talk to your lawyer or consult and hire one. You have listed no facts here at all. How do we know where the mother is? Whoever is revising the children is going to get child support. Just because you have joint custody does not mean you have equal time with them.
IF YOU FOUND THIS ANSWER "Helpful" or " The Best Answer" YOU CAN THANK ATTORNEY RADDATZ BY MARKING IT SO because Avvo awards the attorney points. MS. RADDATZ is donating her time and talent by answering questions to help those in need of legal information. This is NOT a consultation and in no way creates an attorney-client relationship. YOU SHOULD ALWAYS PERSONALLY CONSULT WITH AN ATTORNEY IN YOUR LOCAL AREA who has specific expertise in the area of law you are asking about. Remember that it is the Lawyer who: Punishes the wicked, Protects the innocent, Raises up the lowly, Opposes brutality and injustice, Seeks equality of humanity regardless of color, cast, sex or religion, Leads in every cause, and Seeks the best in everything.