Yes. Custody and visitation are mutually exclusive. Custody is the legal right to make decisions regarding a child. Visitation is the time each parent spends with the child. Even though the child maybe living with the other parent, you can still be joint custodian. You will need a revision of the joint custody plan and that will probably require a lawyer.
Ms. Harcourt has nailed the distinction on custody and visitation for you. I'm not clear why the other attorney told you something so very different. It is very possible to have joint legal custody and not have equal physically present parenting time. And, while courts will give some weight to where an older teen prefers to live, that continues to be balanced on what is in the best interests of the child.
Absurd allegations, at last those that are clearly absurd, tend to help the other party more than the party who is asserting the absurdity, as that party has their credibility diminished in the eyes of the court.
Sign up to receive a 3-part series of useful information and advice about child custody law.