  We propose a new method for exemplar-free class incremental training of ViTs.
The main challenge of exemplar-free continual learning is maintaining
plasticity of the learner without causing catastrophic forgetting of previously
learned tasks. This is often achieved via exemplar replay which can help
recalibrate previous task classifiers to the feature drift which occurs when
learning new tasks. Exemplar replay, however, comes at the cost of retaining
samples from previous tasks which for many applications may not be possible. To
address the problem of continual ViT training, we first propose gated
class-attention to minimize the drift in the final ViT transformer block. This
mask-based gating is applied to class-attention mechanism of the last
transformer block and strongly regulates the weights crucial for previous
tasks. Importantly, gated class-attention does not require the task-ID during
inference, which distinguishes it from other parameter isolation methods.
Secondly, we propose a new method of feature drift compensation that
accommodates feature drift in the backbone when learning new tasks. The
combination of gated class-attention and cascaded feature drift compensation
allows for plasticity towards new tasks while limiting forgetting of previous
ones. Extensive experiments performed on CIFAR-100, Tiny-ImageNet and
ImageNet100 demonstrate that our exemplar-free method obtains competitive
results when compared to rehearsal based ViT methods.
