Was it the lawyer's closing fee alone or does the amount about which you are complaining include all of the other costs incurred at closing: filing fees for the notice of Settlement, deed and mortgage (if any), which can be $300-$400; postage, title insurance, mortgage pre-paids and associated fees, taxes and wire transfer fees are all paid by the buyer at closing. Without knowing what the flat fee that you were originally quoted and without seeing exactly what you were charged, it is impossible to tell whether or not your attorney's charges were proper, but everything should be spelled out in your HUD-1.
I agree with the last commentator that all of the details of your transaction should be on the HUD-1 Settlement Statement. However, I would ask the attorney for a detailed invoice, showing all elements that went into the fee.
Technically, a written retainer agreement is not required in a real estate matter in NJ; but your lawyer should have confirmed his fee by letter or some other writing. If your recollection of your conversation with the lawyer about his fee at the start of the transaction is correct, it is not appropriate to increase the fee by such a large amount without good reason and disclosure.
There are fee arbitration committees in each county of the state. I have included a link to the page of the NJ Judiciary website explaining more about fee arbitration. It is much cheaper than filing in court, and will probably make your lawyer a little more "cooperative" about dealing with your fee.
Please consult an attorney directly before taking action. This answer is intended for general information only and should not be taken as legal advice. My communication with you is not privileged and is not within, or intended to create, an attorney-client relationship. Pursuant to Circular 230 of the Department of Treasury: (1) no written statement to be provided by me relating to any Federal tax transaction or matter is intended to be used, and no such statement can be used, for the purpose of avoiding penalties that may be imposed on the taxpayer, and (2) such written statement may not be used by any person to support the promotion or marketing of or to recommend any Federal tax transaction or matter.
As the other comments suggest, it is very difficult to pinpoint exactly what fees you are complaining about, but if you feel you have been overcharged, contact the local bar association for assistance.