You do not tell us how much earnest money was involved, but probably more than a consultation with an attorney before having let it go. First, if the purchase agreement has a mediation and/or arbitration clause, that must be followed in any effort to recover the money. Otherwise, you may sue in the court in the county where the seller resides. The fact that you signed the agreement releasing the money under bad advice of your fiduciary (agent) is a negative but not fatal circumstance. Again, this is all about economics and whether or not there is sufficient value to expend the time, money and effort to recover your deposits.
You may wish to consult a real estate lawyer for advice but it sounds like you have an uphill battle at this point, especially since you signed the agreement to release the earnest money. The real question would be whether you had good cause to back out which from what you said may be speculative itself. Again, probably best that you find a real estate lawyer and seek some advice.
Advice on this forum is for informational purposes only and should never be mistaken as a substitute for legal advice. If you are in need of legal advice, you should consult local legal counsel.