If the lot is part of an existing subdivision (is it developed?) you may need to work out something with the other owner.s Are you sure you don't want to be in the subdivision? What do you want the land for, and is it something not allowed in the subdivision? Definitely you will need a local land use attorney to walk you through (or maybe out) of this.
This response is not to be construed as establishing an attorney-client relationship, and provides general information on the subject at hand only.
I do not practice in your state, so would suggest you speak to a local residentia real estate attorney. Here, where there is a subdivision developer, they usually have the right to withdraw parcels from the subdivision while they are in control of the Association (if an HOA) and as long as that parcel of land is withdrawn before too high a percentage is sold to others, otherwise, the developer cannot do it without consent from the other owners who may have purchased on the belief that that parcel was part of the subdivision. Why would it matter? Because they may have believed costs would be shared amongst a larger percentage of owners. In some instances, you not only need the consents of the other owners, but also all mortgage lenders, who also have an interest in the value of homes where they have lent funds. It can be an arduous process, and it may be better not to proceed with the deal if you may be caught in something you were not aware of. You need to check all your paper work to confirm whether you were misinformed or just not personally aware. Again, seek advice from a local attorney - the cost for an hour's consultation is way cheaper than thousands lost if this isn't the right real estate for you.
This is not intended to be legal advice or create an attorney-client relationship. If more information is needed, you should consult with an attorney in your state regarding the specifics of your situation and the options available to you.