You should consult with an attorney, licensed in your jurisdiction, who is experienced in handling matters in this area of law. A skilled attorney in your jurisdiction will know which questions to ask you about your situation in order to assist you with making an appropriate determination regarding your situation.
This is not legal advice. This response is provided for general information only, as a public service. It is not to be relied upon as legal advice and does not create an attorney/client relationship; nor is it an attempt to create an attorney/client relationship. Consult with local counsel in your jurisdiction about the specifics of your case, which is the only way to gain true meaningful legal guidance and/or representation.
Yes, the father can and will ask to have some sort of custodial and visitation privileges with your son. That's because in California, the more time you spend with your child, the less child support you pay. Down here in Orange County, judges are increasingly favoring an equal timeshare for parents. In your case, a judge would likely order frequent, shorter visits until your son got used to his dad, and then dramatically increase the visitation time within a couple of months. In order to find out how the judges in your area tend to rule, contact a very experienced family law attorney before you decide how to proceed. By the way, under the Probate Code, your son's father has abandoned him and you have a possible action to terminate his parental rights. If you do this, however, you can never ask for child support. Good luck.
Sign up to receive a 3-part series of useful information and advice about child custody law.