Very short phrases cannot be copyrighted, but they can be the subject of trademark protection. The more likely case is that you were accused of trademark violations rather than copyright violations. If so, then it sounds like EL James intends to try and get trademark protection for those very-identifiable phrases, which is definitely a possibility.
That said, to your specific question can the word "GREY" be protected, probably not. But that is not what you said you were selling. If you want to sell T-shirts with nothing on them but the word GREY, you should be fine.
The whole book is no doubt copyrighted, and you can be liable for copyright infringement for using a phrase from a copyrighted book.
But not every use of phrases from a copyrighted work is a copyright infringement.
The question, in your case, is whether it would be a fair use, and subject to the fair use defense, which is not a simple question in your case.
If you have the money for litigation, it would be an interesting case to take on.
You are probably infringing trademarks. Phrases such as this can be protected by trademark law. Thus, the removal of your items was probably appropriate. Further. you may face a suit for trademark infringement. If you have not yet done so, you need to retain intellectual property counsel. You probably should attempt to avoid litigation by negotiating a settlement with the trademark owner.
More generally, if you make tee shirts, then you need to work closely with intellectual property counsel before you place phrases, designs and photographs on them. Any time you try to profit by using famous names, photographs, or associating yourself with famous celebrities, you are probably violating intellectual property rights unless you have a license or written permission.
The relevant law is: “Any person who, on or in connection with any goods or services, … uses in commerce any … term … which … is likely to cause confusion, or to cause mistake, or to deceive as to the affiliation, connection, or association of such person with another person, or as to the origin, sponsorship, or approval of his or her goods, services, or commercial activities by another person … shall be liable in a civil action by any person who believes that he or she is or is likely to be damaged by such act.” 15 U.S.C. 1125.
So what E.L. James is asserting is that the display on your tee shirts of certain terms taken from her books is likely to cause tee shirt consumers who have read her books to falsely believe that your shirts are sold by her or that she sponsors or endorses your sales or that you or your shirts are somehow affiliated, connected or associated with her and her book.
Because, and only because, her book trilogy has sold millions of copies she likely has a valid argument. Each of the terms you use, however, must evaluated separately because while one could be well-known to her readers others may not. For those other many more millions of us who have never read her books we have no idea what terms could reasonably be associated with her – exclusively associated with her, that is. If some term was not coined by her then she has no grounds to prevent you from putting it on a tee shirt.
At the end of the day, however, only your own attorney can evaluate all the facts and provide you with meaningful advice.
The above is general information ONLY and is not legal advice, does not form an attorney-client relationship, and should NOT be relied upon to take or refrain from taking any action. I am not your attorney. You should seek the advice of competent counsel before taking any action related to your inquiry.