You may want to review your facts and options with an attorney.
If what you wrote is what the father is saying, he may be trying to trick you into signing custody of the child over to him.
A parent does not need to be a custodial parent to claim most tax deductions and credits relating to the child. The IRS has a form that the custodial parent can sign to document that the custodial parent is releasing any right to claim tax deductions and credits in favor of the noncustodial parent for the specified year.
The Child and Dependent Care Credit is special in that the noncustodial parent cannot claim this credit even if it is his year to claim the child as a dependent.
Depending on how much money is involved, you two may come up with some agreement to offset the Child and Dependent Care Credit.