Without court orders, you and the father each have equal rights to custody. Generally, you can pick up your son from anyone other than his father. You cannot remove him from his father's custody without his father's consent.
Ultimately, you need to get custody orders from a court, otherwise you face a battle of possession with you and the father taking custody of your son any time he is with someone else.
You can reach Harkess & Salter LLC at (303) 531-5380 or info@Harkess-Salter.com. Stephen Harkess is an attorney licensed in the state and federal courts of Colorado. This answer is for general information only and does not create an attorney client relationship between Stephen Harkess or Harkess & Salter LLC and any person. You should schedule a consultation with an attorney to discuss the specifics of your legal issues.