  Utilizing well-trained representations in transfer learning often results in
superior performance and faster convergence compared to training from scratch.
However, even if such good representations are transferred, a model can easily
overfit the limited training dataset and lose the valuable properties of the
transferred representations. This phenomenon is more severe in ViT due to its
low inductive bias. Through experimental analysis using attention maps in ViT,
we observe that the rich representations deteriorate when trained on a small
dataset. Motivated by this finding, we propose a novel and simple
regularization method for ViT called Guided Transfer of spatial Attention
(GTA). Our proposed method regularizes the self-attention maps between the
source and target models. A target model can fully exploit the knowledge
related to object localization properties through this explicit regularization.
Our experimental results show that the proposed GTA consistently improves the
accuracy across five benchmark datasets especially when the number of training
data is small.
