If he has never filed a paternity action and there is no court order regarding time sharing, then you are the legal guardian, and visitation is up to your discretion. However, if you refuse altogether to allow him to see her and he does file a paternity action, your refusal is one factor the court consider that could be harmful to you. Your primary consideration in making this determination should be your child's physical safety and her psychological health. If she exhibits behavioral problems in relation to her father's behavior, you may want to take her to a play therapist for evaluation.
If you found this answer helpful, let me know by clicking the "Mark as Helpful" button at the bottom of this answer. It’s easy and appreciated.
Sign up to receive a 3-part series of useful information and advice about child custody law.