ORS 96.060 partially addresses this issue. Under ORS 90.060:
(1) When any person has built or builds, by mistake and in good faith, a fence on the land of another, such person or the successor in interest of the person may, within one year from the time of discovering the mistake, go upon the land of the other person and remove the fence, doing no unnecessary damage thereby.
(2) The occupant or owner of land whereon a fence has been built by mistake shall not throw down or in any manner disturb such fence during the period which the person who built it is authorized by subsection (1) of this section to remove it.
So, assuming your neighbor made a "mistake" in "good faith" you would need to notify the neighbor of the mistake, demand they remove the fence, and then wait a year before removing it yourself. It would be even better if you called the county surveyor out to confirm the fence was on your land and provide the neighbor with documentation from the surveyor. If it wasn't a "mistake in "good faith" by your neighbor you may be able to remove it now, but realize that you're taking additional risks, and you'd better be right about the boundary line. Even if you are right about the boundary line, your neighbor still could claim that it was a "mistake" in "good faith," particularly if they can blame it on the fencing contractor. If you do remove the fence, it is a good idea to be as careful as possible removing it and to save the fence materials for the neighbor.
You are right to be concerned about the property line. Your neighbor could end up acquiring part of your property through adverse possession after 10 years if certain conditions are met. In the end, recognize that self-help is risky and unforgiving if you are wrong. It is best to hire an attorney to sort out these issues through the legal process.