  Although transformer networks are recently employed in various vision tasks
with outperforming performance, extensive training data and a lengthy training
time are required to train a model to disregard an inductive bias. Using
trainable links between the channel-wise spatial attention of a pre-trained
Convolutional Neural Network (CNN) and the attention head of Vision
Transformers (ViT), we present a regularization technique to improve the
training efficiency of ViT. The trainable links are referred to as the
attention augmentation module, which is trained simultaneously with ViT,
boosting the training of ViT and allowing it to avoid the overfitting issue
caused by a lack of data. From the trained attention augmentation module, we
can extract the relevant relationship between each CNN activation map and each
ViT attention head, and based on this, we also propose an advanced attention
augmentation module. Consequently, even with a small amount of data, the
suggested method considerably improves the performance of ViT while achieving
faster convergence during training.
