You do need to notify the government ANYTIME your address changes. Whenever you do so, be sure to use a method that gives you delivery confirmation and save your delivery confirmation. It does happen some times that the government mistakenly sends letters to the wrong address. If you present your delivery confirmation you will be able to reverse any negative consequences caused by the government's mistake.
It is very important that you notify the government of any address change. It is quite simple to notify them. Go to USCIS.gov. Select forms on the top left and it should be the first form available. Simply fill out and send. Same building or not, if the unit is numbered different from your previous unit (Unit A to Unit B for example) then you will need to submit a change of address form.