  Learning new classes without forgetting is crucial for real-world
applications for a classification model. Vision Transformers (ViT) recently
achieve remarkable performance in Class Incremental Learning (CIL). Previous
works mainly focus on block design and model expansion for ViTs. However, in
this paper, we find that when the ViT is incrementally trained, the attention
layers gradually lose concentration on local features. We call this interesting
phenomenon as \emph{Locality Degradation} in ViTs for CIL. Since the low-level
local information is crucial to the transferability of the representation, it
is beneficial to preserve the locality in attention layers. In this paper, we
encourage the model to preserve more local information as the training
procedure goes on and devise a Locality-Preserved Attention (LPA) layer to
emphasize the importance of local features. Specifically, we incorporate the
local information directly into the vanilla attention and control the initial
gradients of the vanilla attention by weighting it with a small initial value.
Extensive experiments show that the representations facilitated by LPA capture
more low-level general information which is easier to transfer to follow-up
tasks. The improved model gets consistently better performance on CIFAR100 and
ImageNet100.
