  Continual learning refers to a dynamical framework in which a model receives
a stream of non-stationary data over time and must adapt to new data while
preserving previously acquired knowledge. Unluckily, neural networks fail to
meet these two desiderata, incurring the so-called catastrophic forgetting
phenomenon. Whereas a vast array of strategies have been proposed to attenuate
forgetting in the computer vision domain, for speech-related tasks, on the
other hand, there is a dearth of works. In this paper, we consider the joint
use of rehearsal and knowledge distillation (KD) approaches for spoken language
understanding under a class-incremental learning scenario. We report on
multiple KD combinations at different levels in the network, showing that
combining feature-level and predictions-level KDs leads to the best results.
Finally, we provide an ablation study on the effect of the size of the
rehearsal memory that corroborates the efficacy of our approach for
low-resource devices.
