  Replay-based methods have proved their effectiveness on online continual
learning by rehearsing past samples from an auxiliary memory. With many efforts
made on improving training schemes based on the memory, however, the
information carried by each sample in the memory remains under-investigated.
Under circumstances with restricted storage space, the informativeness of the
memory becomes critical for effective replay. Although some works design
specific strategies to select representative samples, by only employing a small
number of original images, the storage space is still not well utilized. To
this end, we propose to Summarize the knowledge from the Stream Data (SSD) into
more informative samples by distilling the training characteristics of real
images. Through maintaining the consistency of training gradients and
relationship to the past tasks, the summarized samples are more representative
for the stream data compared to the original images. Extensive experiments are
conducted on multiple online continual learning benchmarks to support that the
proposed SSD method significantly enhances the replay effects. We demonstrate
that with limited extra computational overhead, SSD provides more than 3%
accuracy boost for sequential CIFAR-100 under extremely restricted memory
buffer. Code in https://github.com/vimar-gu/SSD.
