This is a very delicate situation, and of course, you need to contact an immigration attorney that can help you with any and all issues that may come up. Most attorneys offer free consultations, whether they be via phone or Skype—and that’s something you would definitely benefit from. There are several unknown things here that are very important, for example, you say "visa" but by that do you mean a "Green Card"? What paperwork have you filed on his behalf? Did you file an I-130, Petition for Alien Relative, and has that been approved? Is he waiting on "Consular Processing" to get an "immigrant visa"? Chances are, that if you are at the beginning stages of said immigration proceedings, the process can be stopped since you may not "stay with him." I'd suggest you talk to an immigration attorney to tell you if the "Sponsor Contract" you have with the US Government reasuring that your spouse will not become a public charge, and that you'll reimburse the government if he does, is still valid (and if your spouse will be able to immigrate to the US).