While you might legally remove the fence without a lawsuit, a lawsuit is the safest way if your neighbor won't agree. If you have had a surveyor come out, and you are certain that the fence is on your property, you could start to remove the fence without a law suit. What probably happens next is the neighbor calls the police, if you haven't finished removing the fence they will tell you to stop, and if you have removed part of the fence, they cite you for damage to property. The police won't get involved in refereeing a property line dispute. They issue a citation and let the criminal courts sort it out. You are much better off getting a civil court to order the removal of the fence as opposed to being forced to appear in a criminal setting having to prove it was your property. While you might try the case on your own, it would need to be in superior court, and you are much better off talking to an attorney who could represent you.
This answer is for general purposes only, and it does not create an attorney-client relationship.
You might also check with your local zoning/code enforcement department.
In some areas, a permit is required to install a fence. A survey/sketch showing the proposed fence on the right property is generally required. If a permit was required but not obtained, or if the permit was violated by putting the fence on your property, the City or County might force the neighbor to remove it.