See my answer to your other post.
I will also add: if you can show that your ex is not the one interested in visitation with the child and that he's really only exercising visitation with the child to take the child to the paternal grandma, the court could take that into consideration when allocating visitation time. Also, if the grandma is an alcoholic and poses a danger to the child, if you ex does ask for visitation time you could ask the court to put restrictions on his visitation time so that he weren't allowed to take the child near the grandma.
You should consult with a family law attorney.
The choice of a lawyer is an important decision and should not be based solely upon advertisements. Any answers to questions are intended to provide general guidance and do not create an attorney-client relationship, and are not intended to provide specific legal advice. You should consult an attorney before relying upon any answers provided.