It's always hard to tell from a few lines just how strong someone's case is, but it certainly sounds like you have a reason to be talking to the judge about your concerns. As you know, it takes a substantial change of circumstances and a finding by the court that it is in the child's best interest in order to get custody modified, but you have described evidence here that, if accurate, could very well lead to a modification. Of course without knowing what the original basis for the court's award of primary physical custody was, there is just no way to be sure. It would be worth a 1-hour meeting with an attorney, with your original order and the Findings and Conclusions, to see if there really is enough to approach the court.
Please bear in mind that providing a response by email does not create an attorney/client relationship. Also be aware that family law, perhaps more than any other area of the law, is often state-specific. I am an attorney in Alaska and speak from that perspective. While I've provided general information to you regarding your inquiry, legal advice should be obtained directly from a practitioner in your state.
Sign up to receive a 3-part series of useful information and advice about child custody law.