Taking a child to a prison to visit an inmate should be unacceptable. If you have "sole custody" then the prison is requiring the permission from you...don't grant it! Tell him that if he wants to visit the prison, send your daughter home.
For the purposes of the prison a parent is a legal guardian. He can take her and does not need your permission.
Another issue is that If you have custody of your daughter then is he making this request for his visitation times? How does she come to be in his presence for this visit? Rather than trying to block such a visit at the prison visitor control center you could try via controls that are built into your parenting plan. Express your concerns to him and see what he says.
If this is his time with your (plural) daughter and he wants to take her there is a good possibility there is nothing you can do. Fighting it might do more damage than allowing her. I have visited people in prison several times and seen children participating in visits. It can be healing for all parties concerned if handled properly.
Sign up to receive a 3-part series of useful information and advice about child custody law.