Check with an attorney about the exact rule in your jurisdiction. Most temporary guardianships are revocable at the will of the grantor and do not trump a divorce dcree that has incorporated a custody arrangement.
Unless the guardianship was court Ordered as part of a dependency or incapacity proceeding, most guardianships can be revoked at will, however, it is preferable to do so in writing. Absent a court order as to the parents respective custody rights, each parent has legal custody rights and physical custody rights that exist under the circumstances you describe are not revoked by recission of a guardianship. It is more likely that you will need to petition your court for a custody order.
Sign up to receive a 3-part series of useful information and advice about child custody law.