  In this paper, we present our submission for the English to Czech Text
Translation Task of IWSLT 2019. Our system aims to study how pre-trained
language models, used as input embeddings, can improve a specialized machine
translation system trained on few data. Therefore, we implemented a
Transformer-based encoder-decoder neural system which is able to use the output
of a pre-trained language model as input embeddings, and we compared its
performance under three configurations: 1) without any pre-trained language
model (constrained), 2) using a language model trained on the monolingual parts
of the allowed English-Czech data (constrained), and 3) using a language model
trained on a large quantity of external monolingual data (unconstrained). We
used BERT as external pre-trained language model (configuration 3), and BERT
architecture for training our own language model (configuration 2). Regarding
the training data, we trained our MT system on a small quantity of parallel
text: one set only consists of the provided MuST-C corpus, and the other set
consists of the MuST-C corpus and the News Commentary corpus from WMT. We
observed that using the external pre-trained BERT improves the scores of our
system by +0.8 to +1.5 of BLEU on our development set, and +0.97 to +1.94 of
BLEU on the test set. However, using our own language model trained only on the
allowed parallel data seems to improve the machine translation performances
only when the system is trained on the smallest dataset.
