  This paper presents the first Swedish evaluation benchmark for textual
semantic similarity. The benchmark is compiled by simply running the English
STS-B dataset through the Google machine translation API. This paper discusses
potential problems with using such a simple approach to compile a Swedish
evaluation benchmark, including translation errors, vocabulary variation, and
productive compounding. Despite some obvious problems with the resulting
dataset, we use the benchmark to compare the majority of the currently existing
Swedish text representations, demonstrating that native models outperform
multilingual ones, and that simple bag of words performs remarkably well.
