If the Court has already entered a visitation order, then you will have to go back to Court to ask that the visitation be suspended/supervised pending a determination of the father's drug involvement. I believe the Court will immediately order both of you to undergo full drug screening (hair follicle and urinalysis). If he turns up positive (and as far as I know there is no way to "beat" a hair follicle test -- they can use ANY kind of bodily hair), the Court will undoubtedly order supervised visitation. If you can convince the Court that his family is unable/unwilling to supervise properly, there is a private visitation supervising service in Columbia. I would advise you, though, that the Court generally makes the supervising relative a party to the action so they are under a Court order to ensure the child's safety.
Do NOT take it upon yourself to simply deny visitation. This would be YOUR violation of the Court order and even if you think that because you believe the father is doing drugs that justifies withholding visitation, it could backfire. However, if the father appears to pick the child up AND he is visibly under the influence AND, hopefully, you have an independent witness to corroborate your claim, you MIGHT be justified in calling law enforcement.
The best thing you can do at the present time is consult with an attorney as soon as possible and get an emergency/expedited hearing scheduled.
Legal disclaimer: Janne Berry Osborne is an attorney duly licensed to practice law in the State and Federal Courts of South Carolina, including the Bankruptcy Court. This answer is offered for general educational and informational purposes only. It is not offered as, and does not constitute, legal advice nor does it constitute or create an attorney-client relationship between Janne Berry Osborne and any other person. You should schedule a consultation with an attorney to discuss your specific legal issues.
Sign up to receive a 5-part series of useful information and advice about child custody law.