If you have temp orders in place, and he is not following the orders, then you should file a Motion to Enforce.
Failure to comply with the Order of the Court is punishable with 6 months in jail and up to $500 fine for each offense. With your Motion, you should request a hearing.
You must give him 10 days notice of the hearing if you want him held in contempt, or 3 days if you are just seeking to enforce the terms of the Order. At the hearing, present your evidence showing that he has violated it, and ask the Court to rule in your favor. It is also helpful if you bring a proposed order to the hearing.
Contact a local attorney to help you file a motion to enforce access, and ask for a finding of contempt. If you have a caseworker due to CPS being involved, discuss the issue with the caseworker as well.
This answer is general information which does not establish any attorney-client relationship between the person asking a question and the person answering, or a duty to respond to ongoing questions; nor is it intended to replace competent legal assistance in the jurisdiction where the matter/issue arises or is before a Court.
Sign up to receive a 5-part series of useful information and advice about child custody law.