If you can establish that he provides an unsuitable environment for the child, then the judge may decide to give you sole custody. Him masturbating alone while you and your daughter are in another room is not grounds for losing custody (if it was, then 99% of fathers in this country would lose custody), but if you can establish other erratic behavior, then you have a shot. You should consult with a family law attorney about this.
THESE COMMENTS ARE NOT LEGAL ADVICE. They are provided for informational purposes only. Actual legal advice can only be provided after consultation by an attorney licensed in your jurisdiction. Answering this question does not create an attorney-client relationship or otherwise require further consultation.
Sign up to receive a 3-part series of useful information and advice about child custody law.