Get a survey. Then get a real estate attorney you can call if it escalates. Don't threaten to call a lawyer. Call one.
The above is general legal and business analysis. It is not "legal advice" but analysis, and different lawyers may analyse this matter differently, especially if there are additional facts not reflected in the question. I am not your attorney until retained by a written retainer agreement signed by both of us. I am only licensed in California. See also avvo.com terms and conditions item 9, incorporated as if it was reprinted here.
As Mr. Doland correctly stated, get a survey and a lawyer. You are going to need a survey done anyway. You may be surprised at the results. The concern that you would have is a lawsuit for adverse possession or prescriptive easement. Because of that, it is critical that with every conversation or writing that you stand by the fact that you consented to the fence location. That will help destroy the "adverse" use element of both causes of action.
If you start this process with an attorney, you will make less mistakes along the way.
Kevin A. Spainhour, Esq. - Hopefully this information is helpful. My answering this question giving my general thoughts does not create an attorney/client relationship and is not a legal opinion. The only way to create an attorney client relationship is to retain our services and that can be done over the phone, email or in person.