  Despite advances in dependency parsing, languages with small treebanks still
present challenges. We assess recent approaches to multilingual contextual word
representations (CWRs), and compare them for crosslingual transfer from a
language with a large treebank to a language with a small or nonexistent
treebank, by sharing parameters between languages in the parser itself. We
experiment with a diverse selection of languages in both simulated and truly
low-resource scenarios, and show that multilingual CWRs greatly facilitate
low-resource dependency parsing even without crosslingual supervision such as
dictionaries or parallel text. Furthermore, we examine the non-contextual part
of the learned language models (which we call a "decontextual probe") to
demonstrate that polyglot language models better encode crosslingual lexical
correspondence compared to aligned monolingual language models. This analysis
provides further evidence that polyglot training is an effective approach to
crosslingual transfer.
