Unfortunately, your question is more complex than can be reasonably answered in an online posting. Typically it is very difficult to force the other spouse out of the family residence absent either domestic violence or some kind of pattern of ongoing harassment, threats, etc.
As much as you may not want to hear this, you're going to have to find a job. In the aftermath of separation and/or divorce, a family's lifestyle is devastated and must change/adapt to the new circumstances. In California, each parent has a legal obligation to support his/her children to the best of his/her ability, and each spouse has the obligation to utilize his/her best efforts to become/stay self-supporting. There is no legal right, per se, in this state to be a stay-at-home parent.
Unless divorce papers have been filed (by you) or served on you (by him), there are no orders preventing you from moving to Kentucky with your kids. However, if your husband gets wind of your desire, he may attempt to obtain a Court order preventing your "moveaway" with the kids to another state. At least initially, there's a good chance he may be successful at this, but I don't know enough about your situation to state that with certainty.
Once you move, he may try to obtain an order compelling you to return the kids back to California, given that this has been their home for more than the past 5 years. The moveaway laws in California have evolved over the past 10 years, and the Court's are reluctant to permit a parent to moveaway if the other parent can show a detriment to his/her parent-child relationship because of the move. If that is shown, the Court must then make a determination about what is in the "best interests of the children," which almost always will require a custody evaluation. You may want to read the case "In re Marriage of LaMusga" (if you google it, you'll come across lots of online postings of it).
It doesn't matter whether you have filed for divorce or not. You do not have the right to move to another state and take the children without a court order. Given the situation you might be able to get one but why give up the family home. With a good lawyer and a correctly filled in divorce petition and custody order you can force this guy to move out. Based on what you've said he has no right to full custody and probably doesn't want it. Don't let this bum threaten you into giving up your rights.
You really need to find a local family law attorney to talk to. Your question is very complex and requires the attention of an attorney sitting across from you or at least on the phone and not over the internet. You might want to rely on that Kentucky family to provide the funds to hire an attorney that can assist you through the process. Good Luck.