Custody can be dealt with in The Superior Court and in Probate Court. The later tends to be cheaper. I assume you are merely talking about temporary custody which can be arranged through the state department of children and families, which would essentially be free. Adoption is more complicated and requires the consent of both parents, even the felon, and a resulting court order. Simply "signing over custody" is not possible. It is not illegal for a parent to leave a child with a responsible adult. Parents do this all the time. This DOES NOT change custody which remains with the responsible parent who retains "custody" for all legal purposes.
Sign up to receive a 5-part series of useful information and advice about child custody law.