I would encourage you to speak with local counsel about your options. It sounds like you are contemplating buying the water front property, at this time. Perhaps, you should have a contract contingency that gives you a certain amount of time to evaluate the dock issue with the prospective neighbors. I would try to negotiate with the neighbors about how to resolve the issue. You may need to pay them to relocate their docks (if possible). If you cannot resolve the issue within the contingency time period, then the Purchase and Sale Agreement would terminate. If you go ahead and buy the property without resolving the issue, then you may need to resort to litigation if the neighbors are uncooperative. I do not handle much litigation anymore. However, I was often asked "how much" a given case would cost. Litigation can be rather expensive with an uncertain outcome and is the last resort in my opinion. There are a number of variables which can impact cost, namely: nature and complexity of the claims being litigated; discovery expenses (i.e.; depositions, interrogatories, etc..); expert witness expenses; and, most importantly, the parties themselves. I had a breach of contract real estate case where the parties (my client included) despised each other and were not going to settle. When it was all said and done, my client's fees and cost were about $70,000 for what ended up as a 3 day bench trial (no jury). The only winners in that case were the attorneys when the case should have been settled. Again, speak with counsel first about your options as a prospective buyer here. If you buy the property and this issue remains, you could spend lots of time and money trying to resolve the issue without certainty.
This answer is merely for general informational purposes only and does not constitute an attorney/ client relationship. You are strongly advised to consult with an attorney licensed in your home jurisdiction to get detailed advice on your legal matter based on the laws of your home jurisdiction and case facts.
I like Mr. Dunaway's suggestions for how to approach this.
I would add that it is really impossible to predict the potential litigation costs in a dispute like this with any reasonable accuracy, because it largely depends on how aggressively the defendants fight the lawsuit. Something like this could easily cost you well over $100,000 in attorney's fees and other costs.