Has he sued you for custody? If so, pay close attention to the allegations he has made in the complaint. Additionally, are you operating with or without a custody order? If you do not have a custody order, then no one technically has custody of your son, and he could theoretically keep your son until he decides to give him back. I would pursue a custody order to set your mind at ease.
I agree with the previous posts. If there is no custody order than you aren't "allowing" him to see your son. He can easily just decide to keep your son and not return him until there is an agreement. I reccomend filing for custody immediately and getting even an agreed order on record. I assume he is looking for something more solid so that his visits with your son are not dictated by your schedule? If you guys get along well, then you will be able to agree to a custody stipulation and go about your day. If you guys don't agree, then you will need to go through the entire custody process.
If you found this answer helpful, let me know by clicking the "Mark as Helpful" button at the bottom of this answer. It’s easy and appreciated. This answer does not create an attorney/client relationship and is for informational purposes only. More importantly, the information contained in this answer should not be relied on without first consulting with an attorney who practices in the related area in the related jurisdiction.