You have not made your main desire clear in your question. Are you interested in suing him for money, or are you interested in getting back the joint custody? If the former, I would say you have a weak legal case. The judge, right or wrong, is the one who apparently authorized him to keep your son, and you will have a hard time suing the judge. My advice is that you should ignore the already spilt milk and focus on the more substantive battle over future custody. Get a good lawyer, move quickly, and focus on what you can realistically accomplish.
You can certainly file your request with the court to achieve your custody and parenting goals. From the sounds of it you have a real legal mess on your hands. Do yourself a favor and retains experienced family law attorney to assist you.
This comment is designed for general information only, and should not be construed to be formal legal advice nor the formation of a lawyer/client relationship.
Generally, agreements (even notarized ones) regarding child custody, visitation, and child support are not enforceable--only the order of a court is. This is why it is so important to consult an experienced family law attorney in your area when making these arrangements. Good luck!
Nothing contained here is meant, in any way, to establish an attorney-client relationship and this information is a matter of opinion only, in the nature of general legal information and is not meant to be relied upon, in any capacity, by any individual or entity.