Is it illegal for a credit card company to allow access by phone into my account to someone other than me?

My cc company said a guy called on my account, saying he was my family member. This guy is a stranger, pretending to be related to me so that he could increase my credit line. He ended up purchasing $5,000 of items after he gained access. How come the company allowed him to make changes and purchases? Doesn't it have to be the actual card holder the representative speaks to and not anyone else?