If the HOA gave you implied or express permission to build and now they're changing their minds after the fact, you have some protection, but in general HOAs can be pushy about these things. The HOA declarations and bylaws need to be evaluated, along with the validity of HOA governance proceedings, to answer your question more definitively.
This answer is intended as a courtesy only, and does not constitute an attorney-client relationship between the attorney and the questioner.
If the CCRs authorize the HOA board to take enforcement action and your privacy panel violates the CCRs, then most likely the board can do what it is threatening to do. If you want to stop them, you should immediately hire an attorney who can seek a TRO and preliminary injunction to preserve the status quo. Be forewarned that going done this road will be an extremely costly process--you will end up spending thousands in attorney's fees. So you should think seriously about how much that privacy panel is really worth to you. It is rarely the case that an individual homeowner gets their way in these types of disputes.