You'd benefit from a consultation with a local family law attorney. The last lines in the parenting plan signed by the Court are something like "the failure of one parent to follow the terms of this order are not a basis for the other parent to fail to follow the terms of this order". In other words, two wrongs don't make it right.
I agree with you that it is important to have contact information for your ex and your children when they are with him.
If you don't think he's been following the plan correctly, ask the Court to find him in contempt, don't exercise self-help and cut him off from the kids. If he wants to go to mediation (probably because your plan requres it) then go. if you refuse that is grounds for you to be found in contempt.
His failures as a husband do not mean he's a bad father. Your kids need you to not be in jail and futhermore your kids need you and their father to get along well enough to parent them. The WA pattern parenting plan omits a number of very useful clauses that should be there to eliminate ambiguities and resolve problems. This could be a super opportunity to reach some agreements that will resolve your issues with your ex.
Hope this helps. I know it is not what you wanted to hear, but . . . Elizabeth Powell
A weekly guide with tips and legal advice for each stage of the process.