If you think that there remain hazardous materials on the property from the previous owner that either violate the law or constitute a breach of contract you probably should consult with a California environmental attorney. You haven't told us why this is now a problem so many years later and after the county approved a cleanup, so it is hard to comment further.
Your property insurer will not likely represent you. However, you may be able to accomplish getting the situation returned to the way it was before sim;ly by calling the state or federal EPA and reporting what was done.
This is really not an environmental/natural resource topic, but I'll venture an answer anyway. An LLC is inexpensive and easy to set up. I think moving the real estate into one would be fairly strait forward too. However, I don't see the need for a corporate structure as long as you carry adequate insurance on the property, including liability coverage.
Illinois and the federal government do not ban open fires in the context you are talking about. I don't know whether Palatine does, apparently not from your story. If legal in your town, there is little you can do about it from a regulatory perspective. You always have actions for personal injury available to you but would have to prove negligence.
I am not a Georgia lawyer but you would need to know the permit conditions that were made a part of his permit. If he is not violating any conditions your best approach may be complaining to the office that issues the permit about the problems it is causing.
You absolutely do need an environmental attorney. Neither the bank approval nor a missed issue by your consultant save you. Current owners are liable. You may have an action available against your consultant if they negligently missed this issue. You don't say where the property is located. You should contact a good environmental attorney in that state.