  We critically evaluate the widespread assumption that deep learning NLP
models do not require lemmatized input. To test this, we trained versions of
contextualised word embedding ELMo models on raw tokenized corpora and on the
corpora with word tokens replaced by their lemmas. Then, these models were
evaluated on the word sense disambiguation task. This was done for the English
and Russian languages.
  The experiments showed that while lemmatization is indeed not necessary for
English, the situation is different for Russian. It seems that for
rich-morphology languages, using lemmatized training and testing data yields
small but consistent improvements: at least for word sense disambiguation. This
means that the decisions about text pre-processing before training ELMo should
consider the linguistic nature of the language in question.
