I understand your predicament. You may be able to draft an agreement to give over custody to her of your child so long as she will no longer seek child support but sometimes the Courts will not permit an agreement to be enforced. Your child probably would benefit with your involvement and with having visitation with you even though she has made it very difficult. You probably will continue to have to deal with these issues with her for quite a while unless the agreement regarding custody and visitation is drafted in such a way to address theses issues. While I can't offer you specific legal advise without looking at the documents involved, it may be worthwhile to consult an attorney in the area where the case is being litigated.
In New Jersey, the right of child support belongs to the child not the parent receiving support. Therefore, signing away your rights would not eradicate your obligation to pay support. While the mother of your child might agree to forego child support in a written document, if she changed her mind a year or two from now and wanted support, that agreement would not be enforceable. You would be obligated to pay child support. Also, usually NJ courts require someone to step into your shoes (i.e. a step parent) before it will permit parental rights to be voluntarily signed away.
No - outside of the context of an adoption or DYFS action, you can't voluntarily terminate parental rights / responsibilities in NJ.
There's an often-cited trial level case on the issue where the dad went through what you're describing and worse: http://scholar.google.com/scholar_case?case=12666217077188208034&q=rh+mk+new+jersey+a+sometime+thing&hl=en&as_sdt=4,31 . The court has been clear that "Parental rights are not a sometime thing. The termination of those rights entails a permanent sundering of parental ties...." "The termination of parental rights is governed strictly by statute. ... [T]he Supreme Court held that the termination of parental rights by consent of the parties is valid only when effected through voluntary surrender of a child to an approved agency or to the New Jersey Division of Youth and Family Services (DYFS) accompanied by a formal document acknowledging termination of parental rights, or through a private placement adoption."
I have seen matters where the parties do this "de facto" - the CP agrees to CS of $1 / week (citing and departing from the guidelines) and "all parenting time will be by consent of the parties" (and none is agreed), but the door is always open on both sides to go back. I don't know whether your ex would agree to this or not.
If you decide to pursue it, I suggest getting in touch with pro se support groups like Fathers And Children's Equality http://www.facenj.org (mostly central and south Jersey) and the New Jersey Council for Children's Rights http://www.njccr.org . You'll find many others in those groups who have been exactly where you are.