If you obtained legal custody as a result of dad's parental rights having been terminated, then he has no say. You could seek to have the child's mom appointed guardian, which would allow the child to reside with her and allow mom to make educational decisions and act in your place, etc. Of course, that will require court involvement.
You are going to need court involvement. And, because of so many variables that your post does not address, you are going to need to hire an attorney to review your matter and plan options for you and with you.
In complicated situations like this, it is always best to use the court system than an informal agreement of the parties. You can reach that agreement outside of court, then hire an attorney to formalize it amongst the parties in Court.
In my experience, when stuff like this is done outside of Court it ends up getting messy. Define this as much as you can with a court order, agreed or not.
If this answer gave you the information that you needed, let me know by clicking on the thumbs up below.
[DISCLAIMER- This information is provided as general information only and should not be construed as specific legal advice as this website is not intended to provide anyone with specific legal advice. This attorney is only actively licensed in Illinois. In addition, the answer given above serves for educational purposes only and provides general information and a basic understanding of the applicable law. Anyone using the site expressly consents that there is no attorney client privilege between any person and any attorney responding. No attorney-client relationship is created with the furnishing of this information. This answer should not be used as a substitute for fully discussing the factual nuances of your case with a licensed professional attorney in the state where you are located.]
Sign up to receive a 3-part series of useful information and advice about child custody law.