This answer may depend upon whether the inspector can cite facts to justify not granting the building permit on your refusal to grant access to the house. Your argument would be that a government official cannot make inspection of other parts of your house a condition of granting the permit unless there is justification on the grounds of health and safety. Of course, once he is granted access, he can write up anything he sees, whether or not it is related to the wall damage. You may be forced to appeal the denial to the zoning hearing board and possibly, if you lose, to court. If a client came to me with this question, the first thing I would want to do is to read the ordinance and research case law.
Clifford L Tuttle, Jr
Attorney at Law