If there is a court order indicating you were supposed to initiate contact and you failed to do so, you are in contempt of court. If the court order even impies you were supposed to take action, you could be in trouble.
Frankly, the "no one contacted me" defense is not very convincing. Why did you not follow up? You knew he was entitled to visitation.
Contempt sanctions range from verbal warnings to fines to jail time. You need an attorney to represent and defend you. You cannot care for your children if you are in jail.
I suggest contacting Centro Legal or the Legal Aid Society for assistance/more specific advice.
Centro Legal: http://www.centrolegalwisconsin.org/
Legal Aid: http://www.lasmilwaukee.com/