It's not illegal to have drugs in your system, so you don't face criminal charges for any kind of possession of drugs. However, the State could pursue criminal charges against you on a theory of child abuse/neglect or something similar. It seems to me that it would be a fairly weak case, but they state can file charges on just about anything.
I'm a criminal defense attorney and I do not regularly deal with the Department of Children and Families, which is who I'm assuming the doctor was going to report you to. They could initiate a dependency action against you which could ultimately cause you to lose your children. Once again though, it would be a weak case unless the child was born with some defect or addiction directly related to your alleged drug use.
I practice in Orlando and Orange County even though my office is located in Tampa. Feel free to call me if anything happens or if you would like to discuss these issues further. My number is 813.453.6396.