Mr. Wood is right: you don't have to say, but as you recognize in your question, it's important not to answer in a way that invites the police to hassle you more. Also, under certain state and federal laws, it's illegal to lie to an officer. So it's a good idea to have an innocuous, true response to questions like this already prepared. If you have to think up something on the spot, you may incriminate yourself, or piss off the cop, or tell an illegal lie.
Of your examples, I think "down the road" is the riskiest. "Home" and "work" are pretty safe places for American citizens to admit that they're heading. "Shopping" and "church" are others. This is an interesting question, though, because "I'd rather not say" is likely to get you more hassle than you want.
You can do it the way your lawyer would want. You can say "Officer I know you have a difficult job to do but I respectively decline to answer any questions." Then when you have really pissed him off you can decide what to do next.
If you have nothing to hide, you can avoid a constitutional encounter with the officer and just tell him. I know other defense lawyers will be offended by this alternative but a simple answer to an innocuous question is sometimes simpler than provoking a police officer.