Your neighbors would enforce the restriction. If you bought the 2 acres, and your existing property is contiguous, then most likely you have more than three acres required. If your neighbor would be left with less than three acres, then that property would be in violation of the restrictions.
You can void most deed restrictions with the consent of all interested parties, which would likely include your neighbors. Have you asked them if they would consent to the subdivision and transfer?
In the short term, while you work out the deed restriction issue, you could always get the neighbor to grant you an exclusive easement over the land along with a contract to sell the land to you when you resolve the legal issues. That would give you an enforceable property right with regard to the land now, and time to figure it out.
Yes, you should be able to do this. However, I suggest taking your purchase agreement and the deed (and any other relevant documents) to an attorney with real estate experience so they can review the particulars prior to your signing.