  We propose an algorithm that compresses the critical information of a large
dataset into compact addressable memories. These memories can then be recalled
to quickly re-train a neural network and recover the performance (instead of
storing and re-training on the full original dataset). Building upon the
dataset distillation framework, we make a key observation that a shared common
representation allows for more efficient and effective distillation.
Concretely, we learn a set of bases (aka ``memories'') which are shared between
classes and combined through learned flexible addressing functions to generate
a diverse set of training examples. This leads to several benefits: 1) the size
of compressed data does not necessarily grow linearly with the number of
classes; 2) an overall higher compression rate with more effective distillation
is achieved; and 3) more generalized queries are allowed beyond recalling the
original classes. We demonstrate state-of-the-art results on the dataset
distillation task across six benchmarks, including up to 16.5% and 9.7% in
retained accuracy improvement when distilling CIFAR10 and CIFAR100
respectively. We then leverage our framework to perform continual learning,
achieving state-of-the-art results on four benchmarks, with 23.2% accuracy
improvement on MANY. The code is released on our project webpage
https://github.com/princetonvisualai/RememberThePast-DatasetDistillation.
