Any number of things could cause a parent to be unfit. Drug use, alcohol abuse, neglect of the kids basic, educational neglect, medical neglect, having a lifestyle that is not in the best interests of the children (ie: many significant others brought home, partying too much, working too much), child abuse, domestic violence, committing other offenses that have an effect on the kids, going to jail or prison, etc.
Keep in mind that custody decisions are made on the standard "the best interests of the children." A court does not need to find one parent "unfit" to order a custody change.
Sign up to receive a 3-part series of useful information and advice about child custody law.