With all due respect, you need to adjust your attitude. Your child's mother did not "sign over" her rights to you. She did what was best for your child at the time and that was to decline to fight for custody. She has as much right to parenting time as you do and you have been pretty good about seeing to that.
As for the future, there is no way to anticipate what will happen. You have sole custody. "Letting" the mother see her child is a poor choice of words. As least you do see that the child is "ours" and not "yours." What difference does it make if you have full custody, which relates more to decision making than it does to parenting time? No, your chances do not change unless you become less responsible as a parent. If you are not in fear for your child, it is wonderful of you to agree to more parenting time for the mother. Both mother and child need each other as much as father and child need each other.
Good Lord!! What is it you're interested in, your child, or som eephermeral "rights?" Your mind seems locked in a struggle with yoru ex. Please go back and read your question; it is, essentially: "should my child spend more time with her drug-free Mom . . . or will that hrt my rights?"
What is wrong with you?
Do what is right for your child. Try to put your own interests a little bit behind your child's.
Sign up to receive a 5-part series of useful information and advice about child custody law.