  Online continual learning (online CL) studies the problem of learning
sequential tasks from an online data stream without task boundaries, aiming to
adapt to new data while alleviating catastrophic forgetting on the past tasks.
This paper proposes a framework Contrastive Vision Transformer (CVT), which
designs a focal contrastive learning strategy based on a transformer
architecture, to achieve a better stability-plasticity trade-off for online CL.
Specifically, we design a new external attention mechanism for online CL that
implicitly captures previous tasks' information. Besides, CVT contains
learnable focuses for each class, which could accumulate the knowledge of
previous classes to alleviate forgetting. Based on the learnable focuses, we
design a focal contrastive loss to rebalance contrastive learning between new
and past classes and consolidate previously learned representations. Moreover,
CVT contains a dual-classifier structure for decoupling learning current
classes and balancing all observed classes. The extensive experimental results
show that our approach achieves state-of-the-art performance with even fewer
parameters on online CL benchmarks and effectively alleviates the catastrophic
forgetting.
