You should bring this up at the modification so that the Order can be changed such that a face-to-face exchange of information is not required. Perhaps you should suggest that you exchange information through e-mail or some other non-confrontational means of communication.
You should put all pertinent information into a book and ask that he record what happens on his time an you record what happens on your time and you can hand that to him at the exchange so he can see what has happened and keep a copy for your own records.
I agree that you should ask your attorney to amend your request for modification to include a change to the language requiring an in person exchange of information. In the meantime, you should send him an email indicating that you prefer to exchange information in this manner so that you are both certain all information is exchanged correctly. Do not indicate that you wish to do so because of his hostility or you will engage in a debate about it. There are also several online parent schedulers that assist with exchanging information and dates such as the parenting wizard. Check out that option as well.
The suggestions made are intended to inform and not advise and are based upon general statements of FL laws and specific events or facts may alter the law. You should contact an experienced FL family law attorney for specific legal advice regarding your issue.