  Even with the latest developments in deep learning and large-scale language
modeling, the task of machine translation (MT) of low-resource languages
remains a challenge. Neural MT systems can be trained in an unsupervised way
without any translation resources but the quality lags behind, especially in
truly low-resource conditions. We propose a training strategy that relies on
pseudo-parallel sentence pairs mined from monolingual corpora in addition to
synthetic sentence pairs back-translated from monolingual corpora. We
experiment with different training schedules and reach an improvement of up to
14.5 BLEU points (English to Ukrainian) over a baseline trained on
back-translated data only.
