Your post prompts a number of practical concerns. Is your mother willing to move? Would she cooperate in having your brother forced out? You mention that she is an enabler. I have dealt with many situations like the one you describe. In most of them, the elder parent rants and raves about all of the horrible things they are forced to put up with. When it comes time to do something about it, they usually refuse, for various reasons. Your mother is familiar with the pattern of her life. Would she want to uproot that?
In spite of your brother's problems, he is a live-in caregiver that your mother may need or want. If he moved out, who would assume that role? Your mother also may get some peace knowing that, in spite of the problems, she is providing her son with a place to live and a roof over his head.
These situations are rarely as simple as they might seem on their face.
Having said this, if you believe that your mother is in physical danger, then you can contact the police and/or file a complaint with the court. There is likely to be a hearing, and if your mother agrees with you, then your brother could be evicted. You might also need to obtain a PPO against your brother so that he does not simply force his way back in.
Either way, you are going to need to agree on some follow-up plans for your mother. You should also discuss her estate planning with her to make sure that she has everything in order.