  In many real-world scenarios, data to train machine learning models becomes
available over time. Unfortunately, these models struggle to continually learn
new concepts without forgetting what has been learnt in the past. This
phenomenon is known as catastrophic forgetting and it is difficult to prevent
due to practical constraints. For instance, the amount of data that can be
stored or the computational resources that can be used might be limited.
Moreover, applications increasingly rely on large pre-trained neural networks,
such as pre-trained Transformers, since the resources or data might not be
available in sufficiently large quantities to practitioners to train the model
from scratch. In this paper, we devise a method to incrementally train a model
on a sequence of tasks using pre-trained Transformers and extending them with
Adapters. Different than the existing approaches, our method is able to scale
to a large number of tasks without significant overhead and allows sharing
information across tasks. On both image and text classification tasks, we
empirically demonstrate that our method maintains a good predictive performance
without retraining the model or increasing the number of model parameters over
time. The resulting model is also significantly faster at inference time
compared to Adapter-based state-of-the-art methods.
