Proabably the officer could, the threshold to justify such a search is not very high.
No, you should not have said yes, knowing you had nothing to hide. Sounds like you handled it just fine. Well done!
READ THIS BEFORE CALLING OR EMAILING ME: I am licensed to practice before the state and federal courts in Virginia. We have not established an attorney-client relationship unless we have a signed representation agreement and you have paid me. I am providing educational instruction only--not legal advice. You should speak with an attorney to whom you have provided all the facts, before you take steps that may impact your legal rights. I am not obligated to answer subsequent emails or phone calls unless you have hired me. I wish you the best of luck with your situation.
The officer could compel you to stay. If he had no reasonable suspicion for your further detention, any evidence of another crime would be suppressed and the additional charges dismissed.
You exercised your right to be free from an unreasonable search and seizure! It worked and you didn't have to spend the money to hire an attorney. You did the right thing.