  Adapters present a promising solution to the catastrophic forgetting problem
in continual learning. However, training independent Adapter modules for every
new task misses an opportunity for cross-task knowledge transfer. We propose
Improvise to Initialize (I2I), a continual learning algorithm that initializes
Adapters for incoming tasks by distilling knowledge from previously-learned
tasks' Adapters. We evaluate I2I on CLiMB, a multimodal continual learning
benchmark, by conducting experiments on sequences of visual question answering
tasks. Adapters trained with I2I consistently achieve better task accuracy than
independently-trained Adapters, demonstrating that our algorithm facilitates
knowledge transfer between task Adapters. I2I also results in better cross-task
knowledge transfer than the state-of-the-art AdapterFusion without incurring
the associated parametric cost.
