Yes, but I would wait until there is an established history that this is his pattern: more than a month, and six months is probably sufficient. A year certainly is.
You would file a petition to modify the parenting plan, on the grounds that the child has been integrated into your household in substantial deviation from the parenting plan. There are many pitfalls, so I recommend that you use an experienced local family law attorney.
If this answer is helpful or the best answer, please say so. Legal disclaimer: In accordance with the Avvo community guidelines, this communication does not constitute "legal advice", nor does it form an attorney-client relationship.
Sign up to receive a 5-part series of useful information and advice about child custody law.