You could be leaving yourself open for him to file a motion to enforce the divorce decree against you. This leaves you open to penalties and fines, but not until you have had a hearing on it. You should also expect that this is what he is setting you up for, based on the timing. In any case, if he files the motion, you will need to retains attorney to defend yourself and possible file a counter motion against him alleging any potential wrongdoing on his part. I would imagine that this process does not seem like something you want to get involved in at this point, so you may want to rethink denying his visitation.
Mr. Smith's answer to a legal question on Avvo does not establish an attorney-client relationship. Shawn H. Smith offers a free initial consultation to discuss your case. Feel free to call his office at 512-576-3380 or visit his website at www.shslawoffice.com for more information about his services.
He can file a motion to enforce and have you held in contempt if you withhold the child. The court could order you to pay his attorney's fees and give him make up time. You have to evaluate whether the risk is worth it and whether the father will actually hire an attorney. If not, then all he can do is blow hot air. If you can show he failed to show up anyway, then you don't have anything to worry about.