I would think the easy answer could be found in your employee handbook, if you have one. If not, then, you might want to let the person asking you that you value your privacy and while you respect "their concern" you would rather not discuss the matter. They are your sick days and vacation days and they usually can be used at your discretion. Perhaps a visit to your human resource department would help?
Generally, you are an at-will employee and your employer can do almost anything other than that which is unlawful. With respect to your sick time, generally you can take it at your discretion. Many employers will have policies and unless unlawful they do need to be followed. For example, you may have to provide a basic reason for using sick time so the employer knows it is not being abused. The employer may be able to ask for a physician certification that you were ill; but the details need not be disclosed. Of course, if you need 3 or more days off and seek to take leave under the Family & Medical Leave Act then the employer has more rights and even the federal forms require a physician certification provide the basics of your condition, how long you need off, and when you are expected to return. As for vacation time, this really is at the discretion of your employer. Unless the employer is completely unreasonable, they can require you to follow policies and procedures.
By asking you about the nature of your visit, your employer is foolishly creating a potential liability for itself, namely, they are creating evidence that they KNOW of your medical condition. If you are fired due to a disability, you can argue that your employer KNEW of your condition, and considered it in your termination decision, because they REQUIRED you to tell them about it. This would help you in the event you are terminated due to treatment for, say, a chronic illness like heart disease, cancer, hepatitis, AIDS or the like.
As an employee at will, it is best to never confront your employer, or refuse to do anything. It is always best to be diplomatic. As most people know, your employer can "get even" with you, and hide its true motives, if you make them angry by "refusing" to comply with their request.