Short and to the point:
I filed for unemployment compensation. I was denied. I filed an appeal. I was approved. The employer filed another appeal to RAAC. I was then denied.
Why should I be liable for the incompetence of the first referee in said appeal? I am constrained to conclude that I am not liable under these circumstances.
Fit to be tied.
You're thinking about this as being "liable for the incompetence" of that decision. Instead, analyze what occurred:
You received benefits that are payable out of the employer's taxes, due to a determination of eligibility. When you were determined ineligible under the appeal, you became the recipient of payments which would not have otherwise been made. You were unjustly enriched (legal term), at the expense of the employer and state. It is not some sort of punishment for wrongdoing, or anything of the sort. It just remains that you were paid money that you were not due, and so they're requesting it back. If you were liable for some sort of deception, there'd likely be damages or interest, not merely a restoration of the original funds.
I understand that it is frustrating to have expected an income and then be expected to repay it, but it is legally sound.
This posting is for informational purposes only. Answering this question does not constitute forming a lawyer/client relationship. For more complete advice, specialized to your situation and locality, consult a lawyer from that state.