  Pre-training a large transformer model on a massive amount of unlabeled data
and fine-tuning it on labeled datasets for diverse downstream tasks has proven
to be a successful strategy, for a variety of vision and natural language
processing tasks. However, direct fine-tuning of the pre-trained model may be
suboptimal if there exist large discrepancies across data domains for
pre-training and fine-tuning. To tackle this issue, several previous studies
have proposed further pre-training strategies, where we continue to pre-train
the model on the target unlabeled dataset before fine-tuning. However, all of
them solely focus on language models and we empirically find that a Vision
Transformer is vulnerable to overfitting as we continue to pretrain the model
on target unlabeled data. In order to tackle this limitation, we propose
self-distillation as a regularization for a further pre-training stage.
Specifically, we first further pre-train the initial pre-trained model on the
target unlabeled data and then consider it as a teacher for self-distillation.
Then we take the same initial pre-trained model as a student and enforce its
hidden representations to be close to those of the teacher while optimizing the
student with a masked auto-encoding objective. We empirically validate the
efficacy of self-distillation on a variety of benchmark datasets for image and
text classification tasks. Experimentally, we show that our proposed method
outperforms all the relevant baselines. Theoretically, we analyze the proposed
method with a simplified model to understand how self-distillation for further
pre-training can potentially help improve the performance of the downstream
tasks.
