Of course it is! Where the children are residing at the time of filing of the divorce will certainly affect the Court's decision as to temporary custody. You should certainly try to get the children in your possession before filing. Also, most courts do not allow parent to live with paramours before the divorce is finalized, and sometimes after. You need to speak with any attorney experienced in family low immediately.
It certainly could effect the outcome in Court. I will be very difficult for you to convince the Court that it is not in the children's best interests to continue to reside with their father after you have relinquished custody to him for the past 4 years. While the Court can consider other factors, at this time you face the hurdle of explaining why you allowed them to stay there if you did not think it was in their best interests. Unless there is something new that makes their present home environment undesireable, you should attempt to get the children into your home prior to filing for divorce.