You need to consult with a reputable attorney that you trust about how best to handle the situation. The accusations you say that your neighbor are making are definitely slander if they are false, and you have the right to bring a lawsuit for any damage that the slanderous remarks may have caused to your reputation in the community. But before you do that, It would be helpful to have statements from the neighbors stating in writing exactly what the neighbor told them and when it happened. You'll need evidence from neighbors to establish exactly what was said and when. Also, be sure to check with your attorney to see what the statute of limitations is for bringing an action for slander. In some jurisdictions (like Texas), you have only one year to file a slander lawsuit, or it is barred by the statute of limitations.
Sign up to receive a 3-part series of useful information and advice about personal injury law.