Did the notice provide any explanation as to the purpose for the inspections? Is there an issue that has arisen? It sounds like you are on the right track to question their authority to inspect a unit. You might make contact with an attorney in your area to represent you in the event the HOA decides to pursue the inspection, and to advise you re your rights and obligations.
Assuming this is a condominium, Section 718.111(5) Florida Statutes Provides:
(5) RIGHT OF ACCESS TO UNITS.--The association has the irrevocable right of access to each unit during reasonable hours, when necessary for the maintenance, repair, or replacement of any common elements or of any portion of a unit to be maintained by the association pursuant to the declaration or as necessary to prevent damage to the common elements or to a unit or units.
You will notice that end of the provision broadens their scope on the right of access. That can include inspecting units to make sure that the owners aren't keeping anything inside them that could cause damage to other units. "The association" includes board members or their agents. That said, they must advise you of when they intend to enter the unit, and you can be there if you choose.