  Video-based action recognition is one of the most popular topics in computer
vision. With recent advances of selfsupervised video representation learning
approaches, action recognition usually follows a two-stage training framework,
i.e., self-supervised pre-training on large-scale unlabeled sets and transfer
learning on a downstream labeled set. However, catastrophic forgetting of the
pre-trained knowledge becomes the main issue in the downstream transfer
learning of action recognition, resulting in a sub-optimal solution. In this
paper, to alleviate the above issue, we propose a novel transfer learning
approach that combines self-distillation in fine-tuning to preserve knowledge
from the pre-trained model learned from the large-scale dataset. Specifically,
we fix the encoder from the last epoch as the teacher model to guide the
training of the encoder from the current epoch in the transfer learning. With
such a simple yet effective learning strategy, we outperform state-of-the-art
methods on widely used UCF101 and HMDB51 datasets in action recognition task.
