For the purposes of this response, I am assuming that the "mediation agreement" was incoporated into an actual custody order that was entered by a court in Virginia.
FOr starters, you can't force him to discuss changes if he is unwilling to do so and you can't stop him from filing whatever it is he thinks he is going to file with the court (enforcement? modification? change of custody?). Here is what you can do:
1. If you are willing and able to follow the terms of custody and visitation as they are specifically set out in your mediated agreement (but these are not the de facto terms under which you both have been operating), then you can send him a letter by registered mail notifying him of your intent to follow the terms of the mediation agreement (as incorporated into the court order dated "x") effective as of "x" date, due to whatever reasons you may want to set forth in the letter, and enclosing a copy of the agreement and court order.
If the agreement has not been ratified by and/or incorporated into a court order, then you may need to petition the appropriate court to either make an initial custody determination (if no petition for custody is pending) or to enter the agreement into a final order (if there is a petition still pending).
If the agreement is part of a court order and the other parent refuses to comply with the terms (after you've sent notice of your intent to enforce those terms), then you can file a motion to enforce the order or a rule to show cause to enforce the order.
2. If you are either unwilling or unable to follow the terms set forth in the mediated agreement, then you will either need to
(a) petition the court for an initial determination of custody & visitation (the court will consider the mediated custody agreement, but is not required to accept and/or follow those terms) -- this is the proper process to follow if NO petition is pending already and NO order has previously been entered by the court; OR
(b) petition the court for a modification of visitation based on a material change in circumstances having occurred since the date of entry of the current order. The specific facts of your individual situation will determine what facts to allege as a "material change in circumstnaces warranting modification". From what you've described, it could perhaps be your move, a change in the child's schedule due to school, dad's work schedule (if that has changed), inability to communicate with the other parent, or whatever other facts make the terms of the mediated agreement unworkable now.
This response does not create an attorney-client relationship and is intended for general information purposes only.
So far, the two of you are just talking and getting into arguments. That's not working. You need to get a mediator involved. If you worked it out once before through mediation, isn't there reason to think you could work it out again?
Sorry to say, but it sounds as if you are being rather high handed in making demands that you know ahead of time he probably can't meet. If he works nights and is picking her up in the mornings, that means you are asking him to turn his entire schedule upside down just because that's what suits you now. What about your daughter's best interest? What about his schedule? What about just nothing more than the way you decided it ahead of time, without asking what might work for him?
I hope the two of you can work things out. If he won't agree to mediate, however, then you can either live with the uncertainty of not knowing if he is planning to file papers, or you can file for court action yourself.