You need to go in and modify the visitation. Yes courts do sometimes allow jail visitation. And no you do not have to bring the child to the grandmother unless she has a specific grandparent visitation order.
IF YOU FOUND THIS ANSWER "Helpful" or " The Best Answer" YOU CAN THANK ATTORNEY RADDATZ BY MARKING IT SO because Avvo awards the attorney points. MS. RADDATZ is donating her time and talent by answering questions to help those in need of legal information. This is NOT a consultation and in no way creates an attorney-client relationship. YOU SHOULD ALWAYS PERSONALLY CONSULT WITH AN ATTORNEY IN YOUR LOCAL AREA who has specific expertise in the area of law you are asking about.
I agree with attorney Raddatz. However, if the children want to see the grandmother, even though there is no court order in place allowing her visitation... maybe a little time would not hurt. If you feel its in their best interest. She did supervise and has probably gotten used to seeing them... so again, only if you feel it is in their best interest. (Not legal advise... just two cents) :-)