  Scarcity of parallel sentence-pairs poses a significant hurdle for training
high-quality Neural Machine Translation (NMT) models in bilingually
low-resource scenarios. A standard approach is transfer learning, which
involves taking a model trained on a high-resource language-pair and
fine-tuning it on the data of the low-resource MT condition of interest.
However, it is not clear generally which high-resource language-pair offers the
best transfer learning for the target MT setting. Furthermore, different
transferred models may have complementary semantic and/or syntactic strengths,
hence using only one model may be sub-optimal. In this paper, we tackle this
problem using knowledge distillation, where we propose to distill the knowledge
of ensemble of teacher models to a single student model. As the quality of
these teacher models varies, we propose an effective adaptive knowledge
distillation approach to dynamically adjust the contribution of the teacher
models during the distillation process. Experiments on transferring from a
collection of six language pairs from IWSLT to five low-resource language-pairs
from TED Talks demonstrate the effectiveness of our approach, achieving up to
+0.9 BLEU score improvement compared to strong baselines.
