If the child's father will not give permission you will have to petition the court for an order to allow out of the country trips and to obtain a passport for your son. You will need to file a complaint for modification of your current orders. Since this does not sound like an emergency it may take eight months or better to get a judgment, unless the father relents and you reach an agreement.
If your sons father will not give you permission, you will need to petition the court for permission to bring your son to Brazil.
If you wish to discuss this matter further, please feel free to contact me directly.
Sign up to receive a 5-part series of useful information and advice about child custody law.