  Despite the subjective nature of semantic textual similarity (STS) and
pervasive disagreements in STS annotation, existing benchmarks have used
averaged human ratings as the gold standard. Averaging masks the true
distribution of human opinions on examples of low agreement, and prevents
models from capturing the semantic vagueness that the individual ratings
represent. In this work, we introduce USTS, the first Uncertainty-aware STS
dataset with ~15,000 Chinese sentence pairs and 150,000 labels, to study
collective human opinions in STS. Analysis reveals that neither a scalar nor a
single Gaussian fits a set of observed judgements adequately. We further show
that current STS models cannot capture the variance caused by human
disagreement on individual instances, but rather reflect the predictive
confidence over the aggregate dataset.
