The issue likely boils down to whether you need your job more than your company needs you.
If you are an hourly employee, there may be an issue of whether your employer must pay you when calling you outside of business hours.
How often must the company call you? If the company often calls you about business, is the company willing to provide you with a company-paid cell phone and pay you to answer the phone?
Why does the boss want your phone number? If it is to make personal calls, he likely is treading on a sexual harassment claim.
You likely are an at-will employee. This means both you and the employer may stop working with each other at any time for any reason (except for a reason specifically prohibited by law).
Having your home address likely is not enough in some situations. For example, if you are the only one who has the password to some file or application that is needed right away, having a phone number to call you would likely be useful.
Does this boss have a supervisor? Have you talked with that supervisor?