He was keeping her while I work, but when it came down to bring her home I wouldn't. He would only do this out of spite to make me mad.
The answer is no, unless Father has established his legal paternity by an Affidavit of Parentage or an Order of Filiation entered by a Court. Father has no established or enforceable rights of parenting time without a proper court order. You should hire counsel to advance your legal rights both as to support and to specify custody and parenting time rights.