Given that your husband was the legal (even though he wasn't the biological) father of the child, I seriously doub the agency failed to get your ex-husband's agreement to terminate his rights. And why has he waited three years? What does he want to do, take this probably-happy three-year-old from loving parents? Show your paperwork to a lawyer, but, for a number if reasons, you'll almost certainly hear that you can't sue the agency.
The contents of this answer should be considered friendly advice, not legal advice (I'm a pretty friendly guy), and the answer should not be construed to constitute an attorney-client relationship. If you'd like actual legal advice, call me for a free consultation at 813-635-0222. Also, if you liked this answer as much as my big ego thinks you did, be sure to click the thumbs-up button!
Nothing. He never had any rights because your husband was the legal father of the baby. He would get nowhere. This one was on you. No one could force you to give your child up for adoption. You had to consent of your own volition. Had the birth father been contacted at the time of the birth your husband could have had his paternity disestablished and the birth father's established, but three years after the fact it is too late and there is nothing that can be done. I'm terribly sorry. The good news is that because of the choice that you did make, which I am sure you did because your felt it was best based on your circumstances, your daughter is safely placed in a loving home with parents who have wanted her since birth. So while there is much tragedy in your story, there is also much love and hope.
If you found this answer helpful, let me know by clicking the "Mark as Helpful" button at the bottom of this answer. It’s easy and appreciated.
Sign up to receive a 5-part series of useful information and advice about child custody law.