  A common evaluation practice in the vector space models (VSMs) literature is
to measure the models' ability to predict human judgments about lexical
semantic relations between word pairs. Most existing evaluation sets, however,
consist of scores collected for English word pairs only, ignoring the potential
impact of the judgment language in which word pairs are presented on the human
scores. In this paper we translate two prominent evaluation sets, wordsim353
(association) and SimLex999 (similarity), from English to Italian, German and
Russian and collect scores for each dataset from crowdworkers fluent in its
language. Our analysis reveals that human judgments are strongly impacted by
the judgment language. Moreover, we show that the predictions of monolingual
VSMs do not necessarily best correlate with human judgments made with the
language used for model training, suggesting that models and humans are
affected differently by the language they use when making semantic judgments.
Finally, we show that in a large number of setups, multilingual VSM combination
results in improved correlations with human judgments, suggesting that
multilingualism may partially compensate for the judgment language effect on
human judgments.
