  Transfer learning is a simple and powerful method that can be used to boost
model performance of low-resource neural machine translation (NMT). Existing
transfer learning methods for NMT are static, which simply transfer knowledge
from a parent model to a child model once via parameter initialization. In this
paper, we propose a novel transfer learning method for NMT, namely ConsistTL,
which can continuously transfer knowledge from the parent model during the
training of the child model. Specifically, for each training instance of the
child model, ConsistTL constructs the semantically-equivalent instance for the
parent model and encourages prediction consistency between the parent and child
for this instance, which is equivalent to the child model learning each
instance under the guidance of the parent model. Experimental results on five
low-resource NMT tasks demonstrate that ConsistTL results in significant
improvements over strong transfer learning baselines, with a gain up to 1.7
BLEU over the existing back-translation model on the widely-used WMT17
Turkish-English benchmark. Further analysis reveals that ConsistTL can improve
the inference calibration of the child model. Code and scripts are freely
available at https://github.com/NLP2CT/ConsistTL.
