This is a tough situation you're in. You're required to monitor your accounts and report fraud immediately so that the issuing bank on the card can act. Read through your card statements in the fine print for contact information on regulatory agencies that complaints can be reported to. Prepare a simple, straightforward letter explaining what happened, what you did to address it, and why you feel the card company and/or the bank is doing the wrong thing. It will take you time to get an answer. In the meantime, appeal to the card company's and bank's business sense - get a different card and open an account at a different bank, and call to close both. Explain to both representatives on the phone why you are doing what you are doing and see whether they will then be pushed to do something about it to keep you as a customer. The agents that are in charge of closing accounts have a lot of latitude to do things to keep you. I'm fairly certain they will not just credit you back the $6k, but they may push forward an investigation. After doing all of this and maintaining records for same, if all else fails, bring the records to a local attorney and ask whether you may pursue a claim against either.
Any answer provided by me does not create an attorney client relationship with the author of the question. An attorney client relationship can be formed with me only after a consultation and subsequent execution of a retainer agreement or letter of engagement. Notwithstanding that, the prospective client is afforded the protections of the attorney-client privilege before and during the consultation regardless of whether the retainer agreement or letter of engagement is subsequently signed.