  Self-supervised learning (SSL) has made remarkable progress in visual
representation learning. Some studies combine SSL with knowledge distillation
(SSL-KD) to boost the representation learning performance of small models. In
this study, we propose a Multi-mode Online Knowledge Distillation method (MOKD)
to boost self-supervised visual representation learning. Different from
existing SSL-KD methods that transfer knowledge from a static pre-trained
teacher to a student, in MOKD, two different models learn collaboratively in a
self-supervised manner. Specifically, MOKD consists of two distillation modes:
self-distillation and cross-distillation modes. Among them, self-distillation
performs self-supervised learning for each model independently, while
cross-distillation realizes knowledge interaction between different models. In
cross-distillation, a cross-attention feature search strategy is proposed to
enhance the semantic feature alignment between different models. As a result,
the two models can absorb knowledge from each other to boost their
representation learning performance. Extensive experimental results on
different backbones and datasets demonstrate that two heterogeneous models can
benefit from MOKD and outperform their independently trained baseline. In
addition, MOKD also outperforms existing SSL-KD methods for both the student
and teacher models.
