  Self-supervised learning has witnessed great progress in vision and NLP;
recently, it also attracted much attention to various medical imaging
modalities such as X-ray, CT, and MRI. Existing methods mostly focus on
building new pretext self-supervision tasks such as reconstruction,
orientation, and masking identification according to the properties of medical
images. However, the publicly available self-supervision models are not fully
exploited. In this paper, we present a powerful yet efficient self-supervision
framework for surgical video understanding. Our key insight is to distill
knowledge from publicly available models trained on large generic datasets4 to
facilitate the self-supervised learning of surgical videos. To this end, we
first introduce a semantic-preserving training scheme to obtain our teacher
model, which not only contains semantics from the publicly available models,
but also can produce accurate knowledge for surgical data. Besides training
with only contrastive learning, we also introduce a distillation objective to
transfer the rich learned information from the teacher model to self-supervised
learning on surgical data. Extensive experiments on two surgical phase
recognition benchmarks show that our framework can significantly improve the
performance of existing self-supervised learning methods. Notably, our
framework demonstrates a compelling advantage under a low-data regime. Our code
is available at https://github.com/xmed-lab/DistillingSelf.
