He will need to file an application for appointment with the probate court for the jurisdiction where his mother died indicating that she died intestate and requesting official recognition of his status. Assuming no objections, the court will grant him letters of administration.
This answer to this question is very state specific, with that being said however, usually, in an intestate (meaning no Will) situation you may be appointed the Administrator of the decedent's estate by making application to the Probate Clerk in the city or county in which your mother-in-law lived. Please be aware that for your brother-in-law to be appointed the Administrator of his mother's estate, he will likely have to post a bond. That means he must have a credit rating sufficiently high enough to allow for his qualification. There could be further complications with the settling this intestate estate especially if there is real estate involved. I recommend that he have at least an initial consultation with a qualified estate planning attorney before he tries to take on this responsibility.
Assuming that she passed away in the area in which the question was posted, I would be pleased to assist. The first question is whether it is necessary that an executor be appointed. Sometimes, there are options for administering an estate without filing probate. Otherwise, to be appointed, the first steps are various court filings. This is followed by other requirements, such as publishing notice, sending notice, inventorying assets, . . .