I have unfortunately seen this many times. Regardless of whether you were married or not, if he is the biological father, her has equal rights to the child as you. The scenario that you pose seems to be that there is no custody order in place. Without an order, whoever has the child can keep her. He cannot take the child from your possession, and you cannot take the child from his possession. Either one of you can take the child from 3rd parties. I do not condone this snatch mentality, but does he work? If so, can't you get the child from her school or daycare provider? This, of course, assumes that there is no custody order in place. The better way to handle the situation would be to file a motion in the custody action for the Court to set temporary custody/ visitation. Then you wouldn't have this tug-of-war going on. The longer that you go without seeing her, the worse your case is getting. You need to get into court immediately.
The other attorney is correct - this case needs to be heard in court. It doesn't seem like the father has any intention of "being civil" as you have stated you have. It appears as though you need to file a complaint for custody and have the Judge decide custody and visitation. Consult with a local family law attorney immediately.
Sign up to receive a 3-part series of useful information and advice about child custody law.