First of all, I'm concerned that your mediator "suggested" anything, as it is their job to remain objective and not offer their own solution. Mediation does not result in any enforceable agreement unless you signed a consent judgment.
If you and the child's father have a parenting time agreement from the Court and you fail to bring the child (per the rules of the judgment) for parenting time, then the father can file a show/cause petition which would require you to go to Court and explain why you did not provide the child for parenting time. The father can also file stating you are in contempt of the current order.
My suggestion to you would be to find any way to bring the child for parenting time so you do not get in legal trouble. Additionally, the Court will agree that it is in the best interest of the child to have access and a relationship to BOTH parents. If you chose to move out of state and you put the distance between the child and their father, it is your responsibility to continue to uphold the agreement you made.
If he files a motion with the Court, you will have until the Court date to obtain an attorney.
***Answering legal questions is a complex matter and cannot be done without further research. My answer here is based solely on the information given in the question and should be taken as informational only. This is not legal advice. My answer here does not indicate that I represent you now, or that I will represent you at a later time. I take clients on a case-by-case basis, and only after initial consultations. ***
Count yourself lucky that the court allowed you to take th child and move. Its a small price to pay to be required to bring the child back for parenting time..
I have a client that was allowed to move to Italy.. but was required to provide 2 plane tickets for the child annually.. now that is expensive..
Please note that I answered this question with general knowlege of the law and with limited disclosure from you, my answer does not create an attorney-client relationship and I cannot be held responsible for how you, the asker, use the information provided. If you want to contact me directly and retain my services you may through my listing on this site or at 248-901-0750 or firstname.lastname@example.org.