Whether your child's mother is unfit is for a judge to determine, and the standard that is applied in custody matters is what is in the best interest of the child. Leaving the child with her grandmother by itself may not rise to the level of unfit. The analysis you are seeking is very complex and I would highly recommend that you speak with an attorney in person. There are many qualified attorneys on Avvo and anyone at my office would be happy to speak with you.
This answer is for informational purposes only and cannot be construed as creating an attorney/client relationship.