I'm sorry you and your sister are having to go through this on top of losing your mom. I don't know what the problem is with your brother (he may just be a jerk in general), but keep in mind that just like some people are lactose-intolerant, some are simply incapable of dealing with kids, and there's no pill you can take for that. Or maybe this has something to do with your sister's adolescence--which can be a difficult time under the best of ircumstances. It's unfortunate that your mom made the guardianship decision she did, but the good news is that it's really not that big a deal to fix it. I would recommend you hire an attorney to take care of all the procedural end of things, but if you can't, you may still be able to manage it on your own, especially if your probate court is understanding and helpful. This is not by any means a detailed explanation, but basically you'll need to file a request with the court requesting that your sister's guardianship be transferred to you, and you'll have to take care of making sure that all close family members (not just your brother) get notice, in case someone else wants to get the guardianship. The court will have someone independent look into what the current situation is and whether it appears that making you guardian would be what's best for your sister. Hint: it's a pretty big deal that your brother flipped out and dumped her on your doorstep like that--while he's not technically required to have her living with her (just like he could put her in boarding school), unless he had a pretty good reason and it was all handled in a calm, orderly manner, I don't see the court forgiving him for that if you're a good alternative, especially if your sister is going to say she was miserable with him and detail whatever the problems were.
In fact, come to think of it, that may very well be why he's demanding you return her now. Texas law requires that a guardian submit an annual accounting and report to the court (how she's doing generally, in school, medically, what's going on with her property, if any, and WHERE SHE'S LIVING, among other things). He may be about to have to file that for the year (it would be due within 60 days of whatever the anniversary is of the date he initially assumed guardianship), in which case, he would now find himself between a rock and a hard place, realizing he's going to have to either admit he dumped her on you or lie to the court under oath about all this.
Perhaps there are other issues there as well--if your mom left any funds to take care of her, I wonder what's going on there? The guardian's annual accounting should be very specific as to what's happened with the estate, down to the level of a checking account statement, but with details as to the reasons for each expenditure (11/16/11 $23.12 spent for materials for school project, 11/18/11 $214.65 interest paid from CD, 11/19/11 $15.00 copayment for prescription meds, etc.). That's honestly just beyond a lot of people's abilities, and is something you'd have to deal with also if you assumed guardianship. There's also a bond associated with being a guardian, but usually it's not that expensive, unless your sister's estate is very large.
If you have a decent relationship with your brother in general, I'd recommend at least initially approaching him about this informally, as a "Hey, let's just see if we can fix this" thing. Almost certainly, the court will let you do that without it being that big a deal. A contested transfer, on the other hand, would be more expensive for both of you, prolonged, and more difficult for your sister, who needs to have at least a little bit of security about her future. If he doesn't seem receptive, you can tell him you know about the reporting requirements and ask him what he stated in it or is planning to state in it. He probably doesn't realize he's not literally required by law to have her living with him (though that's certainly the preferred approach). Good luck!