You need to have a NJ probate attorney and possibly a CPA review the documents in question. Generally if the house was not her primary residence for 2 of the last five years and the house increased in value there may be capital gain, but there are various deductions permitted so it may not be much.
If you have not been working with an attorney through the process, it sounds as though you should be. If there are only children who are beneficiaries of the entire estate there should be no NJ inheritance tax, if the estate is over $675,000 there may be a NJ estate tax...but you are well past the date the return should have been filed so you need assistance to get it filed. If your mother had a will the will should say how the estate taxes are paid. As a general rule, the estate would be paying (or reimbursing) for estate expenses to keep the estate going. Regarding capital gains tax, if she died in 2011 the house received "stepped up" basis so it is likely that the capital gains to the beneficiaries are minimal but you again need tax/legal advice to make sure all the accounting is done correctly.
This is not legal advice nor intended to create an attorney-client relationship. The information provided here is informational in nature only. This attorney may not be licensed in the jurisdiction which you have a question about so the answer could be only general in nature. Visit Steve Zelinger's website: http://www.stevenzelinger.com/