I agree with Mr. McLane. A civil eviction process will be necessary, especially if he shows that he is paying rent. The issue is further complicated by the fact that you are married and are now seeking a divorce. I would contact an attorney to move forward on both fronts.
Nothing in the answer provided should be considered legal advice because all cases and facts are different and require a thorough examination of the facts by an attorney before any decisions are made.
Your mother can tell him to leave as the owner of the house. You do not mention any type of domestic violence, threats, etc., that would give rise to you seeking a protection order. Thus, only your mother, as the owner of the house can decide who gets to live in the house.
Your mom, as the owner of the home, (and in the absence of a lease) likely has the ability to require him to leave. However, if you have children, always speak to an attorney when kicking out the other spouse to determine if there would be any implications to child custody issues (aka whether the Court would look poorly upon you for any actions you take). In response to your question, though, your mother does have the ability to make anyone leave her home if there is no lease. If necessary, she can likely obtain the assistance of the local police department in having him removed.