  This work proposes a new method to sequentially train deep neural networks on
multiple tasks without suffering catastrophic forgetting, while endowing it
with the capability to quickly adapt to unseen tasks. Starting from existing
work on network masking (Wortsman et al., 2020), we show that simply learning a
linear combination of a small number of task-specific supermasks (impressions)
on a randomly initialized backbone network is sufficient to both retain
accuracy on previously learned tasks, as well as achieve high accuracy on
unseen tasks. In contrast to previous methods, we do not require to generate
dedicated masks or contexts for each new task, instead leveraging transfer
learning to keep per-task parameter overhead small. Our work illustrates the
power of linearly combining individual impressions, each of which fares poorly
in isolation, to achieve performance comparable to a dedicated mask. Moreover,
even repeated impressions from the same task (homogeneous masks), when
combined, can approach the performance of heterogeneous combinations if
sufficiently many impressions are used. Our approach scales more efficiently
than existing methods, often requiring orders of magnitude fewer parameters and
can function without modification even when task identity is missing. In
addition, in the setting where task labels are not given at inference, our
algorithm gives an often favorable alternative to the one-shot procedure used
by Wortsman et al., 2020. We evaluate our method on a number of well-known
image classification datasets and network architectures.
