In the State of Washington, a statement is slanderous per se, if it “(1) exposes a living person to hatred, contempt, ridicule or obloquy, to deprive him of the benefit of public confidence or social intercourse, or (2) injures him in his business, trade, profession or office.” Caruso v. Local Union No. 690 of Int'l. Bhd. of Teamsters, Chauffeurs, Warehousemen and Helpers of Am., 100 Wn.2d 343, 353, 670 P.2d 240 (1983). In a defamation per se claim, “the plaintiff is not required to produce evidence of actual damage to entitle him to substantial damages, because under such circumstances the law presumes damage.” Arnold v. Nat'l Union of Marine Cooks and Stewards, 44 Wn.2d 183, 187, 265 P.2d 1051 (1954). But, to make that presumption, the plaintiff must show the defendant acted with actual malice, not simple negligence.
The Supreme Court of Washington has held that where allegedly defamatory statements go beyond concrete topics like the charge of a crime, whether a statement rises to libel per se is a question of fact for the jury. The jury would have to consider all the circumstances and then decide whether your co-worker's statements were defamatory and whether they caused you harm in your career and/or personal relationships.
I note that you stated that your co-worker has told people in your office that he "thinks" you are gay. If confronted or sued, he may try to fall back on the defense that he was just stating an opinion and not a fact. However, there can be a blurry line between opinion and fact, and once again it would be up to a jury to determine what kind of effect your co-worker intended to create.
In addition to legal action against your co-worker, you may want to consider a meeting with your supervisors to discuss this disruptive behavior on your co-worker's part and demand that something be done to stop it.