  Semantic textual similarity (STS), a cornerstone task in NLP, measures the
degree of similarity between a pair of sentences, and has broad application in
fields such as information retrieval and natural language understanding.
However, sentence similarity can be inherently ambiguous, depending on the
specific aspect of interest. We resolve this ambiguity by proposing a novel
task called Conditional STS (C-STS) which measures sentences' similarity
conditioned on an feature described in natural language (hereon, condition). As
an example, the similarity between the sentences "The NBA player shoots a
three-pointer." and "A man throws a tennis ball into the air to serve." is
higher for the condition "The motion of the ball" (both upward) and lower for
"The size of the ball" (one large and one small). C-STS's advantages are
two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables
fine-grained language model evaluation through diverse natural language
conditions. We put several state-of-the-art models to the test, and even those
performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS
challenging; all with Spearman correlation scores below 50. To encourage a more
comprehensive evaluation of semantic similarity and natural language
understanding, we make nearly 19K C-STS examples and code available for others
to train and test their models.
