  We present an extensive evaluation of three recently proposed methods for
contextualized embeddings on 89 corpora in 54 languages of the Universal
Dependencies 2.3 in three tasks: POS tagging, lemmatization, and dependency
parsing. Employing the BERT, Flair and ELMo as pretrained embedding inputs in a
strong baseline of UDPipe 2.0, one of the best-performing systems of the CoNLL
2018 Shared Task and an overall winner of the EPE 2018, we present a one-to-one
comparison of the three contextualized word embedding methods, as well as a
comparison with word2vec-like pretrained embeddings and with end-to-end
character-level word embeddings. We report state-of-the-art results in all
three tasks as compared to results on UD 2.2 in the CoNLL 2018 Shared Task.
