A lot of people who "meant no harm" have actually in reality done a great deal of harm in many areas not the least of which is in the area of discrimination. I think that the statement may well be actionable and I recommend that you obtain a full consultation with an employment attorney. Do not delay as there are time deadlines in presenting such claims, requiring early notice or the claim is barred.
This answer is offered for informational purposes only. It is not offered as, and does not constitute, legal advice. Laws vary widely from state to state. You should rely only on the advice given to you during a personal consultation by a local attorney who is thoroughly familiar with state laws and the area of practice in which your concern lies.
I agree with the above advice--you should contact an attorney about your situation as soon as possible.
The response given is not intended to create, nor does it create an ongoing duty to respond to questions. The response does not form an attorney-client relationship, nor is it intended to be anything other than the educated opinion of the author. It should not be relied upon as legal advice. The response given is based upon the limited facts provided by the person asking the question. To the extent additional or different facts exist, the response might possibly change. Attorney is licensed to practice law only in the State of California. Responses are based solely on California law unless stated otherwise.