  We study continual learning in the large scale setting where tasks in the
input sequence are not limited to classification, and the outputs can be of
high dimension. Among multiple state-of-the-art methods, we found vanilla
experience replay (ER) still very competitive in terms of both performance and
scalability, despite its simplicity. However, a degraded performance is
observed for ER with small memory. A further visualization of the feature space
reveals that the intermediate representation undergoes a distributional drift.
While existing methods usually replay only the input-output pairs, we
hypothesize that their regularization effect is inadequate for complex deep
models and diverse tasks with small replay buffer size. Following this
observation, we propose to replay the activation of the intermediate layers in
addition to the input-output pairs. Considering that saving raw activation maps
can dramatically increase memory and compute cost, we propose the Compressed
Activation Replay technique, where compressed representations of layer
activation are saved to the replay buffer. We show that this approach can
achieve superior regularization effect while adding negligible memory overhead
to replay method. Experiments on both the large-scale Taskonomy benchmark with
a diverse set of tasks and standard common datasets (Split-CIFAR and
Split-miniImageNet) demonstrate the effectiveness of the proposed method.
