  Multi-Task Learning (MTL) is widely-accepted in Natural Language Processing
as a standard technique for learning multiple related tasks in one model.
Training an MTL model requires having the training data for all tasks available
at the same time. As systems usually evolve over time, (e.g., to support new
functionalities), adding a new task to an existing MTL model usually requires
retraining the model from scratch on all the tasks and this can be
time-consuming and computationally expensive. Moreover, in some scenarios, the
data used to train the original training may be no longer available, for
example, due to storage or privacy concerns. In this paper, we approach the
problem of incrementally expanding MTL models' capability to solve new tasks
over time by distilling the knowledge of an already trained model on n tasks
into a new one for solving n+1 tasks. To avoid catastrophic forgetting, we
propose to exploit unlabeled data from the same distributions of the old tasks.
Our experiments on publicly available benchmarks show that such a technique
dramatically benefits the distillation by preserving the already acquired
knowledge (i.e., preventing up to 20% performance drops on old tasks) while
obtaining good performance on the incrementally added tasks. Further, we also
show that our approach is beneficial in practical settings by using data from a
leading voice assistant.
