You did not build the driveway nor plant the hedge. As a peace offering you can offer to remove the encroaching driveway. You cannot remove the encroaching portion without his consent because that would be a trespass. If you choose to remove it, get a written agreement before you enter onto the neighbor's property. The probability of your neighbor recovering is slight, but it does exist. Speak to an attorney specializing in real estate issues.
The advice given is general for information purposes only and should not be relied on. You should consult with an attorney of your choice to fully advise you about your legal rights and obligations.
I agree with Mr. Sonkin and you may have a claim for adverse posession and your neighbor may have a claim for ejectment. But you both would be better off in the long run to settle the matter. Now you know what he wants and he knows what you want it is time to sit down with a lawyer and see what your options are to settle this issue without huge legal fees.
Only If and until you and I sign an Agreement for Legal Services, I am not your attorney. These answers are provided for informational and/or novelty purposes