Well, he does not automatically "lose" custody. He will however have a hard time showing the court that it is in the best interests of the children, or that it promotes the children's "stability and continuity" to be placed with him. Custody generally flows in the manner that will preserve the "status quo" unless the children are in danger. You have clearly been the primary caretaker, he has clearly not taken their interests in needing him around when he left. So, the likely outcome is that yes, you will have custody and if he wants it, he will have some reasonable contact, but it is not probable that he would be successful if he tried to get custody.
You should be contacting Department of Child Support Service as soon as possible. They will obtain and enforce an order for child support on your behalf. There is NO fee for their services.
Sign up to receive a 5-part series of useful information and advice about child custody law.