A few comments/answers to your question.
1. A Power of Attorney gives you the same legal authorization as the grantor of the power has, but not more power/authority than they have. So, a POA will allow you to do what your grandson's father is legally authorized to do. If father has physical custody, then the POA from father may be sufficient. If mother also signs the POA, then that will give you same legal authority as both grantors have.
2. Unless there is a custody agreement or court order that specifically provides to the contrary, either parent can delegate "care" of their child to another -- such as a baby sitter, relative, school, etc.
3. The authority being delegated by a POA will last until it expires by the terms set out in the POA (i.e., a limited POA -- which limits either the circumstances or the time period for which the POA is vaild), until revoked, or until the grantor of the POA's death (or incapacity if it is not a "durable" POA)
4. If the other parent is not happy with this arrangement, then that parent would be free to file for a change of custody and, if granted, the POA will be limited to whatever rights the grantor of the POA still has (such as visitation) and you would be bound by the terms of the court order (as the POA)
5. If the father is in active duty military and away on deployment, there are special rules that apply (i.e., there is a statute specifically addressing military parents ability to delegate their custodial/visitation time to family members, etc.)
This response does not create an attorney-client relationship and is intended for general information purposes only.
Sign up to receive a 5-part series of useful information and advice about child custody law.