  While many parallel corpora are not publicly accessible for data copyright,
data privacy and competitive differentiation reasons, trained translation
models are increasingly available on open platforms. In this work, we propose a
method called continual knowledge distillation to take advantage of existing
translation models to improve one model of interest. The basic idea is to
sequentially transfer knowledge from each trained model to the distilled model.
Extensive experiments on Chinese-English and German-English datasets show that
our method achieves significant and consistent improvements over strong
baselines under both homogeneous and heterogeneous trained model settings and
is robust to malicious models.
