Let's assume that you said no, and there were no exigent circumstances. More important than whether it was legal is how you go to prove that it happened. It is unlikely that the officicer will admit that they came in anyways after you said no, and if they did they will claim that they saw something in plain view that constituted the likelihood that a crime was being committed, or that they smelled an odor that indicated that a crime was being committed.
Discuss your case with no one. Take all that were present to an experienced criminal defense lawyer. Hire a lawyer, don't go with the Public Defender. They PD's office is simply too overloaded with cases now days. It was bad before, but since the law change to the Office of Conflict Counsel instead of using outside attorneys for appointments, they are super duper jammed up.
Challenging seach and seizure is time consuming and will require depositions, analysis, research and memo/motion writing. While PD's are certainly qualified to do this, they simply are given way to many cases to allow their best talent on each case.