Technically, yes, but it's not a great idea. You should consult an attorney first. If you pick up and leave in the middle of the night, then he can file a motion and the court might make you do far more of the transportation for visits. Courts prefer to see parents try to work things out before they alter the status quo. But generally speaking, if you're the primary custodial parent, you'll be able to move with the court's blessing.
You can definitely move, but the father of your child may go to Family Law Court and petition for custody. If this happens, the Court may either order you to move back or force you to do most of the grunt work for your father to see your child. See if you and the father can come to an agreement before you leave. This will make it less likely that he will go to Court and petition for official custody and challenge your move.
Sign up to receive a 3-part series of useful information and advice about child custody law.