I have never heard of this happening. You have the right to appeal.
Please be advised my answers to questions does not constitute legal advise and you should not rely on it, due to the fact that we have never met, I have not been aprised of the facts in you case nor have I reviewed any documents.
You may want to consult an attorney about your exact situation.
If you are confident that you have been reporting your income properly and paying the correct taxes, you have nothing to worry about. Still, an attorney can explain whether anything about your work or your reporting of your income raises any potential flags. If the attorney believes you have a legitimate claim and believes you have a legitimate explanation for your work and earnings, he or she will let you know and either offer to represent you or not.
Also, write down whatever you remember about the call center representative: male or female, date and time of call, number called, anything else that stuck about the representative, and what exactly you were told. If you miss a deadline on appealing the claim, it can be helpful to have that sort of information to support a "good cause" statement to allow your claim to move forward.