I left it as it was for the past 20 years as it was on the side access to my property but now it is on the main entrance to my property as I have converted my property. The new path is about 18 inches shorter in width from the middle of the boundary fence to the front of the property. He is flatly refusing to lose part of the 18 inch space between the side of his property and our boundary fence which is his but which I had to pay for when replacing the broken panels. I am having new furniture delivered in May but the space is not wide enough to get the furniture through and he has said I can take the panels down but not move the boundary line back to straight.
The boundary line is not necessarily determined by where the fence is located. If you have not done so, you should have the property surveyed. When the survey is complete, you should have it reviewed by an experienced real estate lawyer in your area. Your lawyer can advise you on the best way to proceed.
These disputes can get complicated, and have a number of issues involved. If you left it alone for twenty years, that could be a problem. When was the last survey? Can you not access your property at all based on what he is saying is the new line? You need to visit a real estate attorney and go over all the facts to make a decision about what to do.
I agree with everyone else, but your question is a bit vague and confusing. If the 18" are under your title, then it's your land. I believe you can still bring an ejectment and quiet title action to recover the 18". However, give the consent, implied or actual as to the placement of the fence, I can see a judge ordering you to replace a fence on the boundary at your cost (usually w/in 6" of the boundary for safety) if 12 is enough, otherwise if you actually need the whole 18" that necessity might either win it for you, or with your neighbor's consent I can see the judge making you put the fence up at your cost just inside his boundary. BAD FENCES MAKE MAD NEIGHBORS.
Property line disputes arise when neighbors can't agree on where the property line lies. They can often be resolved by looking at the deeds to the property.