You can be penalized, but the court taking your son away is pretty harsh and unlikely to happen unless this is something you do all the time. You might get fined or have to give him make-up time, which is reasonable. More likely your husband is just threatening you. If he files, hire an attorney. Otherwise, blow him off and follow the decree going forward.
This answer is intended for informational and educational purposes only and should not be considered legal advice nor forming the attorney client relationship. This attorney is licensed in Texas.
Mr. Hardng's advice is accurate--it's unlikely that your custody arrangement would be changed for that alone. If he's threatening that, appearing to cave in may not be a good idea, but you know the man better than we do: do you think it might help to offer a sincere apology and perhaps say you'll give him the next weekend or holiday of his choice that he otherwise wouldn't get to spend with your son to make up for this?
Sign up to receive a 5-part series of useful information and advice about child custody law.