More information is needed about the entire provision. However, you can specifically exempt camping trips and other group activities where there will be both sexes.
Based on the information you provided, arguably, you did not violate the agreement because the other people were not visitors. A reasonable interpretation of that provision would be no overnight visitors in your residence.
However, I cannot predict what a judge will determine. Therefore, you are correct to modify the agreement to take into consideration group activities.
If you have additional questions, please contact me.
Sign up to receive a 10-part series of useful information and legal advice about the divorce process.