Custody law is usually state-specific and I don't practice in California, but based on my experience in Pennsylvania, I can give you some general information.
First of all, just because a report has been called in, it doesn't mean that CPS will take your child. Given that you are concerned about how the father is treating your child, though, you should think about trying to get custody of your child.
Usually, if there is no custody order or agreement, the law regards both parents as having equal right to have custody of their child.
Most states also require CPS to look at suitable relatives (including the other parent if the parents live separately) for placement of a child before taking the child to foster care.
It might be helpful if you start a custody case now. If the child is living with the potentially abusive father and not you, the CPS worker is going to wonder why. Even if you don't file for custody, you should do everything you can to show that your child would be safe with you - that will help make your case that you are a better resource for your child -- both for the CPS worker and the custody court. If you're in a situation where you don't think you can provide a stable and safe home for your child right away, then you should consider asking a relative that you trust if they would be willing to step in.
Finally, don't avoid the CPS worker -- if you do, they might think you have something to hide. Be polite, but keep in mind the worker is looking out for what he or she thinks is best for your child. This might be completely different from what you think is best for your child, so be careful about what you say -- it may come back to haunt you.
File the motion and speak to CPS. CPS usually does not remove children on a neglect referal. Generally, if the child were to be removed and CPS knew where to find you, they would contact you about placement.