He has no rights unless he establishes a legal relationship with baby
by signing a voluntary acknowledgement of parentage or by court order. Being the biological father isn't enough to confer legal rights.
Until and unless he establishes this critical legal relationship, you are under no legal obligation to bring baby to jail. He would have to establish to the satisfaction of the judge that it would be in an infants best interests to visit with him, a felon, in jail. I think that would be a heavy burden for him. Do what's right for your baby: ignore his family's demands.
The answer to your question depends upon whether the father has signed the Voluntary Acknowledgement of Paternity and the child's birth certificate or whether he has at least registered with the Putative Father registry. It appears you know the man is your daughter's biological father but unless and until he has signed the papers mentioned, or been deemed the father in court, he is not legally the father and therefore has no rights or responsibilities concerning the child. If he has no rights, he is also not liable for child or other financial support.
As for the man's family pushing you to allow him to "claim rights," what exactly does that mean? At this point, nothing. The man's family has no standing to push anything. Go consult with an attorney so that if and when the man files legal papers, you are prepared to move forward. For now, you do not need to do or agree to anything.
Okay, so, he's not just the father, he's also your husband. That changes things. The "rights" you're worried about are defined by a court. If you nor he files papers in divorce court, then no rights will ever be defined.
If, however, you or he files for divorce, then he could ask for visitation rights (and you could seek support). His "rights" would be defined in the court's order.
There are a few ways outside of divorce court that rights could be defined, but they are rarely used.
if he had the multiple felonies before you married him or before you conceived a child with him, you cannot complain about that now. you made those choices knowing who and what he is.
as your husband, this child is legally his. he does have rights that come with that but also responsibilities.
you may ignore his family if you wish. it has no rights yet.
best for you to consult with a lawyer.
Sign up to receive a 3-part series of useful information and advice about child custody law.