I tend to agree with the advice you have received. However, your case is not to be taken lightly (a little worrying is merited under these facts). Your facts could raise concerns for the USCIS regarding misrepresentations on your prior removal of conditions case (did you advise of the separation?) or on your Naturalization case, did you lie about making prior misrepresentations in your current application. (I know it sounds convoluted, but this is actually the thinking of some USCIS offices).
If you want a reliable second opinion, you should schedule a consultation with a second experienced immigration attorney and bring your entire file. If your attorney that is confident in his/her position then they should not have a problem with that.