No. You do not have the "right" to visitation unless it's spelled out in an order. If there is a previous custody order that does not give you visitation, you must go to court to get it changed to provide you time with your children. Nothing voids a visitation agreement except a court.
However, the regular visits you have made will be a good basis for an argument that the children should have visitation with you because they are accustomed to it and the sudden change on the part of the children's mother is not in the children's best interest.