I am assuming that you went to court and received an order issued by the court giving you guardianship. If the order was temporary there is a date for you to return to court for a final determination by the court. If you never went back to court for a final order then the tempory order would have expired at some point. In that event there is no valid order of guardianship. Therefore, you can just return the child to your son.
On the other hand, if there was a final order issued by the court your son needs to file for custody of the child and the mother would have to be served and you can agree to give your son custody. Ultimately, the court would make a best interest finding to decide whether your son should have custody of the child. The more you align yourself with your son and if he is found to be a fit parent the more likely he will get a favorable final order.
Sign up to receive a 5-part series of useful information and advice about child custody law.