He's misinformed. The 12 year old gets to express an opinion in certain circumstances but does not get to choose.
This answer DOES NOT establish an attorney-client relationship. This answer is based on the limited information provided and is not intended to be conclusive advice. There are likely other factors that might influence or change the advice after a more lengthy consultation.
Kids don't get to choose and switch. They are with you until a court orders otherwise. The court may hear the kids opinion if it so chooses. Start looking around for a lawyer you might want to use should he try something and do NOT let him bully you with misinformation.
Kids have a say, but do not get to decide where they live. Get a lawyer today - do not wait for him to file a modification. The lawyer can help you with a plan in advance to help your case. You will be ready for court - and he will not.