  We introduce Rosita, a method to produce multilingual contextual word
representations by training a single language model on text from multiple
languages. Our method combines the advantages of contextual word
representations with those of multilingual representation learning. We produce
language models from dissimilar language pairs (English/Arabic and
English/Chinese) and use them in dependency parsing, semantic role labeling,
and named entity recognition, with comparisons to monolingual and
non-contextual variants. Our results provide further evidence for the benefits
of polyglot learning, in which representations are shared across multiple
languages.
