Self help can be a bit tricky. You have a right to remove materials belonging to someone else from your property. However, you do not have a right to damage their property so you should do it in a manner that does not cause damage. If you do remove the structure, take photos of everything you do, before and after, and place all of the materials on his property (you also do not have a right to possess his property). A strong legal position to have would be to have a court order to allow you to remove the structure. You may want to retain a local attorney so you can address this appropriatly.
While self-help in certain situations may be permissible, it also may not be a good idea. If the structure has been there for an extended period of time, (usually 5-25 years depending on the state you’re in), your neighbor may have gained some rights to that section of your property through adverse possession. Even if it hasn’t reached the point of adverse possession, taking matters into your own hands could lead to violent confrontations. If your assumptions about who owns the property turn out to be wrong, moving the structure could result in you being held liable for damaging your neighbor’s property.
That said, if your assumptions about property ownership and the dangerous condition of the structure are correct, your neighbor could be liable for trespass as well as the creation of a private nuisance. I certainly recommend that you contact an attorney with expertise in property matters.