Child visitiation problems and question regarding attorney's seemingly threatening letter?

My ex had a visitation stipulation set up saying he was responsible for pick ups and drop offs. For 2 years he didn't follow this and I met him halfway. He see's his son 2 weekends/month. My hours at work were severely cut and I said no more meeting, let's follow the papers. He refused to drive 20 miles to get his son, had new papers drawn up, which I will not sign saying he picks him up at my residence and vice versa. B/c I didn't sign right away he canceled visits for a full month and now his attorney sent a letter saying the judge WILL grant it and I will be charged with legal fee's. All I tried to do was follow the original agreement and I left messages for the attorney saying I would compromise, but he claims he has never gotten a message from me. I don't feel in the wrong...opinions?