I always tell people that "Uncle Sam makes an appointment with you, you don't make it with Uncle Sam."
I would not reschedule your interview. There is no guarantee that USCIS will agree to reschedule it and if they do it could delay your case for a long period of time. When you are at your interview they will ask you to update your trip list. The day of your swearing in they will ask you to update it once again. At the interview they don't take your passport or your greencard so you will be able to travel after the interview. Just keep good records of when you travel and you will have no problem.