Is this agreement in writing and filed with a court or is it a court order? If not and you are the parent, there should be no reason you can't retrieve your child. If this remains and issue, hire an attorney and seek custody in court. Where is the other parent?
Only a Court can award custody. If you make an agreement, you can revoke it unless it has been made into a Court Order. No Court Order, go get your son. Remember, that after having the child for 3 years, your mother has standing to file for custody of him and take him from you. Contact a good family law attorney in your area to discuss your options and be ready for her.
Good luck to you.
Sign up to receive a 3-part series of useful information and advice about child custody law.