You are exactly right. As long as you did not sign a contract for a restrictive covenant (such as an agreement not to compete with your former employer), and as long as you did not agree to protect your employer's trade secrets, customer lists, etc., any attempt to stop you from doing business with your former client would constitute an illegal restraint of trade in violation of both federal and New Jersey state law.
It is not a simple issue, however. If your former employer believes you used fraudulent or deceptive devices to lure business away, or otherwise interfered with its business operations, they can still sue you under several theories of law. For example, New Jersey law recognizes a lawsuit based on the "intentional interference with a business or contractual relationship." So this is definitely the kind of thing you should seek legal counsel for. Be sure to act promptly if you've been served with any papers or threatening letters.
Please feel free to contact me. I am the author of a book on U.S. Employment Law, and our firm is based in Marlton/Mount Laurel.
Mark S. Guralnick, Esq.