  Previous work on cross-lingual sequence labeling tasks either requires
parallel data or bridges the two languages through word-byword matching. Such
requirements and assumptions are infeasible for most languages, especially for
languages with large linguistic distances, e.g., English and Chinese. In this
work, we propose a Multilingual Language Model with deep semantic Alignment
(MLMA) to generate language-independent representations for cross-lingual
sequence labeling. Our methods require only monolingual corpora with no
bilingual resources at all and take advantage of deep contextualized
representations. Experimental results show that our approach achieves new
state-of-the-art NER and POS performance across European languages, and is also
effective on distant language pairs such as English and Chinese.
