if we never went to court who has the custody of the child in front of a court

my ex boyfriend and i separated after the birth. When this happened we never perused custody orders. 5 years later he is threatening that he is getting a lawyer for custody because i am moving in with my boyfriend. he has no previsions for the child in his home and when the child is visiting he pawns him off on his grandparents consistently. he is doing this out of spite not for the well being of the child. I guess what i am asking is if we already have split custody of the child because there has not been a court order yet?