The child's marital status doesn't effect it. The rights of children born in wedlock and born out of wedlock are equal.
So, if the child has lived in NJ for 5 years (or is under age 5), then you would need his written consent or a judge's order. If he won't sign, the court will go through the Baures factors (Google it) - you need (1) a good faith reason for the move and (2) to prove the move wouldn't otherwise be harmful to the child.
If the two of you can't reach an agreement, the court will send you to mediation to determine the issue before a hearing is scheduled. Good luck with it.
Generally, you will need the cosent of the other parent or a court order permitting you to move out of NJ with the child. I am assuming the father also live in NJ. The court will be interested in where you are moving, why you want to move and how you plan on maintianing the father's relationship/contact with the child. It makes no difference whether you were married or not.
New Jersey requires either mutual consent or a court order before a custodial parent is permitted to move out of state given the facts you've set forth above. If mutual consent isn't there, then you have to file a motion with the court. That motion requires you to meet a preliminary test (usually a pretty low bar). After that preliminary test is met, a plenary hearing is scheduled where evidence is submitted on 12-factors, including the reason for the move and evidence that the move won't negatively impact the non-custodial parent's relationship with the child.
One of the things that comes into play is the non-custodial parent's use of awarded parenting time. The current state of the law says that a non-custodial parent who doesn't take advantage of the awarded parenting time has no standing to challenge the fact that moving away will diminish the relationship with the child.
Hope this is helpful.