  Using a language model (LM) pretrained on two languages with large
monolingual data in order to initialize an unsupervised neural machine
translation (UNMT) system yields state-of-the-art results. When limited data is
available for one language, however, this method leads to poor translations. We
present an effective approach that reuses an LM that is pretrained only on the
high-resource language. The monolingual LM is fine-tuned on both languages and
is then used to initialize a UNMT model. To reuse the pretrained LM, we have to
modify its predefined vocabulary, to account for the new language. We therefore
propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a
competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk)
and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four
translation directions.
