A few things to bear in mind:
One: If there is a custody judgment, then that judgment is enforceable regardless of whether one of the parents has a warrant out for their arrest, or has been convicted of a crime. One does not lose all rights over one's children for committing (much less being accused of) a crime.
Two: If you fail to obey a court order for custody, the other parent has the right to file a motion to enforce the judgment. They can ask the Court to order make-up parenting time for anything you wrongly denied them. One of the factors that courts look at when deciding child custody decisions is, is each parent willing to support an ongoing relationship between the child and the other parent? In general, judges disapprove of parents who withhold their children from their co-parents. Children are not prizes to be fought over.
Three: Your ex-whatever cannot unilaterally "remove custody." To do this, he would have to file a motion with the court, and prove to a judge that a change of custody was in the child's best interests. You would have to do the same thing, to "get the courts to remove his parenting time."
Four: If he's got an outstanding warrant, he is very unlikely to file a motion to enforce or to modify, or to respond to one, because any contact with the court system will get him arrested. And, while it is true that parents with criminal records do not lose all custody rights automatically, it does become a lot harder for them to depict themselves as good parents.
Five: Whether a parent pays their child support obligation or not, does not affect their right to see their child.
Six: There is a limited exception to the rule that you should not try to keep a child away from their other parent. If the other parent has done something that implies that they're a danger to the child, you may have a duty to keep them away, until the charge is resolved.
Seven: You can probably find out what this guy is wanted for, pretty easily. A court database search will show any outstanding Oregon warrants. You can go to your county courthouse and use a public terminal for this - there may be clerks there who can help you. (The court database is an old, arcane system. It is not intuitive or user-friendly at all. Many lawyers have access to this system and know how to use it as well.)
In conclusion, the best course is to consult in private with an attorney in your area. They can help you determine what this guy is wanted for, and help file a motion to modify parenting time if that turns out to be appropriate. They can also handle communications with him for you, so he won't be contacting you directly.