You do not state whether there is a custody and visitation order. if there is one, then you should have a lawyer review it for you to discern whether there is a basis in the order to preclude the access with the grandfather. If there is no basis, then you may wish to file a motion to modify the father's access to prevent visitation around the grandfather.
I strongly suggest that you speak with an attorney.
Please be advised that any information or advice given herein does not constitute an attorney client relationship. Further, before taking any actions or deciding not to take any actions, you should seek counsel of an attorney. This forum cannot provide you with the anyalysis and detail necessary for a full and accurate opinion to be rendered.
You should speak to an experienced family law attorney about obtaining or modifying your custody and visitation orders. The court bases it's orders on the best interests of the child. The court will consider the situation, and has the power to deem certain relatives unsuitable for visitation.
Sign up to receive a 3-part series of useful information and advice about child custody law.