Many people have a complaint similar to yours, however visitation is not something that is terminated by the court unless the visits are no longer in the child's best interests. If the father is not contributing financially for the child, you should file a child support case in family court. The father will most likely be ordered to assist you financially to meet the child's needs.
If the father is not the person enjoying the visits, but is rather leaving the child with other people, you should ask him to stop doing it, and if he continues, you might seek court assistance by filing a petition. The visits are for the father and child, and although he may bring the child to see friends and relatives, he should not be dropping the child off and leaving.