  The size and the computational load of fine-tuning large-scale pre-trained
neural network are becoming two major obstacles in adopting machine learning in
many applications. Continual learning (CL) can serve as a remedy through
enabling knowledge-transfer across sequentially arriving tasks which relaxes
the need to fine-tune all network weights from scratch. However, existing CL
algorithms primarily consider learning unimodal vision-only or language-only
tasks. We develop a transformer-based CL architecture for learning bimodal
vision-and-language tasks based on increasing the number of the learnable
parameters dynamically and using knowledge distillation. The new additional
parameters are used to specialize the network for each task. Our approach
enables sharing information between the tasks while addressing the challenge of
catastrophic forgetting. Our approach is scalable learning to a large number of
tasks because it requires little memory and time overhead. Our model reaches
state-of-the-art performance on challenging vision-and-language tasks.
