  The paper gives an overview of the Russian Semantic Similarity Evaluation
(RUSSE) shared task held in conjunction with the Dialogue 2015 conference.
There exist a lot of comparative studies on semantic similarity, yet no
analysis of such measures was ever performed for the Russian language.
Exploring this problem for the Russian language is even more interesting,
because this language has features, such as rich morphology and free word
order, which make it significantly different from English, German, and other
well-studied languages. We attempt to bridge this gap by proposing a shared
task on the semantic similarity of Russian nouns. Our key contribution is an
evaluation methodology based on four novel benchmark datasets for the Russian
language. Our analysis of the 105 submissions from 19 teams reveals that
successful approaches for English, such as distributional and skip-gram models,
are directly applicable to Russian as well. On the one hand, the best results
in the contest were obtained by sophisticated supervised models that combine
evidence from different sources. On the other hand, completely unsupervised
approaches, such as a skip-gram model estimated on a large-scale corpus, were
able score among the top 5 systems.
