Ask for a copy of his survey, and get one yourself. See an attorney. This brings up many legal issues beyond just the fence.
To the PROSPECTIVE client, please call myself or another attorney for your choice with more detaiils and an appointment. My PRELIMINARY answer to your question(s) is for general purposes and based upon what little information you have conveyed. It is based on such limited information that the general answer should never be relied as a reason for your action or inaction. My response does NOT establish an attorney-client relationship and such may only be established by mutual agreement, and the signing of a written retainer agreement, which will generally require payment for our services, as this is what we do for a living and, just like you, we must get paid for our work.. .