A child cannot just refuse.
The law regarding a child's 'right' to choose which parent to live with is murky, and varies considerably by state and jurisdiction. Although not a standard by any means, many states in the last couple decades have begun to give 'consideration' to a child's declaration of custodial preference when the child reaches the age of twelve or thirteen. The judge is normally given almost unlimited latitude in whether or not he listens to a child and how much weight to give to the child's wishes. In short, there is no specific "age". The judge will decide based on a best interest of the child standard.
Contact a local lawyer to talk specifics.
You might find my Legal Guide helpful "How to Choose A Lawyer For You"
You might find my Legal Guide helpful " What Do I Tell My Lawyer"
No one can know what the record is in the case because online we cannot see your documents. You need a lawyer. Check with a lawyer in your locale to discuss more of the details.
Good luck to you.
NOTE: This answer is made available by the lawyer for educational purposes only. By using or participating in this site you understand that there is no attorney client privilege between you and the attorney responding. This site should not be used as a substitute for competent legal advice from a licensed professional attorney with whom you have established an attorney client relationship and all the privileges that relationship provides. The law changes frequently and varies from jurisdiction to jurisdiction. The information and materials provided are general in nature, and may not apply to a specific factual or legal circumstance described in the question.
Sign up to receive a 5-part series of useful information and advice about child custody law.