No custody agreement in place, but I have the child in my care. Does the paternal father have the right to visitation?

I have had my son in my custody since he was born, after his father and I seperated, we went week on week off from Sunday to Sunday. That stopped working when he was unable to provide for him, and I had to supply everything I stopped the visits. He did not put up a fight about it or go to lawyers. He occasionally called to take him for a night here and there. He had ample opportunity to see his son, in the three months my son remained with his grandmother, while I got my house ready and life situated for us on the other side of the country. He never called her, nor made an effort to see him or pay child support. Now my son is with me, and we are going home for a visit with my family, can he "legally" have access to him? I am located in Alberta, Canada and am scared he will run with him.