  Incremental learning (IL) suffers from catastrophic forgetting of old tasks
when learning new tasks. This can be addressed by replaying previous tasks'
data stored in a memory, which however is usually prone to size limits and
privacy leakage. Recent studies store only class centroids as prototypes and
augment them with Gaussian noises to create synthetic data for replay. However,
they cannot effectively avoid class interference near their margins that leads
to forgetting. Moreover, the injected noises distort the rich structure between
real data and prototypes, hence even detrimental to IL. In this paper, we
propose YONO that You Only Need to replay One condensed prototype per class,
which for the first time can even outperform memory-costly exemplar-replay
methods. To this end, we develop a novel prototype learning method that (1)
searches for more representative prototypes in high-density regions by an
attentional mean-shift algorithm and (2) moves samples in each class to their
prototype to form a compact cluster distant from other classes. Thereby, the
class margins are maximized, which effectively reduces interference causing
future forgetting. In addition, we extend YONO to YONO+, which creates
synthetic replay data by random sampling in the neighborhood of each prototype
in the representation space. We show that the synthetic data can further
improve YONO. Extensive experiments on IL benchmarks demonstrate the advantages
of YONO/YONO+ over existing IL methods in terms of both accuracy and
forgetting.
