If you were not required to file tax returns because you did not earn enough income to reach the minimum, then your naturalization application will not suffer because you didn't file. The key question is whether you ever filed an income tax return as a nonresident, which would be a problem. Also whether you ever failed to file a tax return when you were required to file.
Your application will not be adversely effected if you were not required to file income tax returns. If, however, you worked and just decided not to file income tax returns, then you will have an issue. Additionally, if you ever filed income tax returns as a "non resident" that time period that you considered yourself to be a non-resident will not be counted towards the time needed for naturalization.
DISCLAIMER: This answer is intended to provide general information only. No attorney-client relationship has been created by this posting.