It may look bad if you do not sign the release, but it could be worse if you do. You know what you have said and it could work against you depending on the severity of it all. The opposing counsel may be able to depose your therapist if in fact you waive that privilege. But really, this is where having an experienced attorney in your "corner" is most beneficial. Consult with him or her regarding your concerns. You yet have options available.
There is a patient physiologist privilege so you don’t have to release your medical records. However, since you already told the evaluator why not release the records. It’s not supposed to go against you if you don’t but it gives the appearance that you are hiding something.