Unfortunately, it doesn't have to be your fault. If a bank makes an error and credits money to your account that isn't yours, they are still entitled to take the money back when the problem is discovered even though it wasn't your fault -- it's still not your money. Similarly, if the unemployment office overpaid you, then they can take that money back because you sholdn't have received it in the first place. Essentialy, you got use of the money interest free for awhile, but it's not really your money and you are not entitled to keep it.
You can appeal the determination or seek reconsideration if you believe that the money was not payed to you by mistake. Alternatively, this sort of debt is dischargable in bankruptcy if it was the department's error and there was no fraud on your part.
You can reach Harkess & Salter LLC at (303) 531-5380 or info@Harkess-Salter.com. Stephen Harkess is an attorney licensed in the state and federal courts of Colorado. This answer is for general information only and does not create an attorney client relationship between Stephen Harkess or Harkess & Salter LLC and any person. You should schedule a consultation with an attorney to discuss the specifics of your legal issues.