  We propose an unsupervised method to obtain cross-lingual embeddings without
any parallel data or pre-trained word embeddings. The proposed model, which we
call multilingual neural language models, takes sentences of multiple languages
as an input. The proposed model contains bidirectional LSTMs that perform as
forward and backward language models, and these networks are shared among all
the languages. The other parameters, i.e. word embeddings and linear
transformation between hidden states and outputs, are specific to each
language. The shared LSTMs can capture the common sentence structure among all
languages. Accordingly, word embeddings of each language are mapped into a
common latent space, making it possible to measure the similarity of words
across multiple languages. We evaluate the quality of the cross-lingual word
embeddings on a word alignment task. Our experiments demonstrate that our model
can obtain cross-lingual embeddings of much higher quality than existing
unsupervised models when only a small amount of monolingual data (i.e. 50k
sentences) are available, or the domains of monolingual data are different
across languages.
