While the terms of oral agreements are often difficult to prove, they still bind the parties in many circumstances. So, depending on the substance of your agreement, you may have a cause of action against the neighbor.
Perhaps more importantly, anything the neighbor constructs on your property without your permission could give rise to a claim for adverse possession and result in your loss of part of your property after 10 years (or less). Accordingly, you may need to act now to prevent a future loss.
(Nothing in this answer should be treated as legal advice and this answer does not create any attorney-client relationship)