I am not clear on the details of your situation.
However, most situations like this have possession and access (aka custody and visitation) court orders in place from either a divorce or a CPS court case.
If a person denies visitation as ordered by the court they risk a charge of interference with child custody.
So, this is one of those times when general advice is not sufficient. You need to sit down with a lawyer and discuss this situation in detail and get advice based on the details.
Answers on Avvo are for general information purposes only and should not be relied upon as legal advice. No attorney / client relationship is created by providing this answer. For specific advice about your situation, you should consult a competent attorney of your choosing.
The court's order is the court's order. It stands until modified or superceded. Neither you nor CPS can refuse to obey that order until it is modified or superceded. He gets visitation under the order at the times specified.
I am not intending this to be legal advice, because I don't know the particulars of your situation. Call me if you would like to discuss this or other isues.
Sign up to receive a 3-part series of useful information and advice about child custody law.