I understand your concerns. I believe that a major consideration is whether or not your ex has or is in the process of bonding with the infant. You might discuss a gradual parenting time schedule with your ex with a few hours at a time and add more time as your infant grows. You may also want your ex's parenting time to take place at your home until you are comfortable with your ex's bonding and ability to care for your child. There are books written by child therapists/experts which propose parenting plans based upon the age and development of the child. You may want to check out your local bookstore or an online bookstore for some guidance. I recommend that you meet with an attorney and set-up a parenting plan for you and your ex.
I hope this helps! Good Luck!