Fairness does not enter into it, only legal rules.
It is unclear whether you mean "receiver" or "trustee in bankruptcy". Different rules would apply.
Reduce all the facts to writing and consult an attorney for advice you can rely on. The details will count of a lot.
The above is general legal and business analysis. It is not "legal advice" but analysis, and different lawyers may analyse this matter differently, especially if there are additional facts not reflected in the question. I am not your attorney until retained by a written retainer agreement signed by both of us. I am only licensed in California. See also avvo.com terms and conditions item 9, incorporated as if it was reprinted here.
As unfair as it seems, the nature of a ponzi scheme is that those who got in early received money that was not theirs, but instead was money of other investors in the scheme. You are being asked to return the "ill gotten gains" to a common fund so that all victims of the scheme can be equally reimbursed.
One of several reasons why Mr. Doland is correct that you need to consult an attorney on this is because your post is woefully inadequate to give you any idea of what you can expect. If this is a bankruptcy trustee collecting funds, there are time limits on how far back the Trustee can go to take money that was received by participants. If this is a civil fraud claim, the time limits and extent of your duty to return the money is again different. If this is a restitution order from a criminal court somewhere, it is completely different.
No one will care if you have been made whole from the proceeds you have received. They are trying to treat everyone equally, meaning you will likely end up with far less than you have, unless there are legal defenses to the claim they are making.
Good luck to you.
This answer should not be construed to create any attorney-client relationship. Such a relationship can be formed only through the mutual execution of an attorney-client agreement. The answer given is based on the extremely limited facts provided and the proper course of action might change significantly with the introduction of other facts. All who read this answer should not rely on the answer to govern their conduct. Please seek the advice of competent counsel after disclosing all facts to that attorney. This answer is intended for California residents only. The answering party is only licensed to practice in the State of California.
You may not have to give back money voluntarily. But if the government decides that the distributions you got were the result of fraud, they could come after you in a more formal prosecution. You should talk to an attorney familiar with federal criminal investigations as soon as possible.
What can be done will depend on variety of factors, including who is a "net winner" and "net loser" in the fund. Much will also depend on when the trustee or receiver says the ponzi actually started. You could have legitimate distributions that are mixed up with the tainted ones. If you want to fight it, it's an uphill battle, but you will need an attorney to file the appropriate motions to be heard (assuming that there is an opportunity to do so before the court).
Of course, if you're paying out to the court, you might consider looking at whoever sold this thing to you for contribution. And that adds a different wrinkle to many aspects of your question.
The foregoing is not legal advice nor is it in any manner whatsoever meant to create or impute an attorney/client relationship.