  Contrastive Learning and Masked Image Modelling have demonstrated exceptional
performance on self-supervised representation learning, where Momentum Contrast
(i.e., MoCo) and Masked AutoEncoder (i.e., MAE) are the state-of-the-art,
respectively. In this work, we propose MOMA to distill from pre-trained MoCo
and MAE in a self-supervised manner to collaborate the knowledge from both
paradigms. We introduce three different mechanisms of knowledge transfer in the
propsoed MOMA framework. : (1) Distill pre-trained MoCo to MAE. (2) Distill
pre-trained MAE to MoCo (3) Distill pre-trained MoCo and MAE to a random
initialized student. During the distillation, the teacher and the student are
fed with original inputs and masked inputs, respectively. The learning is
enabled by aligning the normalized representations from the teacher and the
projected representations from the student. This simple design leads to
efficient computation with extremely high mask ratio and dramatically reduced
training epochs, and does not require extra considerations on the distillation
target. The experiments show MOMA delivers compact student models with
comparable performance to existing state-of-the-art methods, combining the
power of both self-supervised learning paradigms. It presents competitive
results against different benchmarks in computer vision. We hope our method
provides an insight on transferring and adapting the knowledge from large-scale
pre-trained models in a computationally efficient way.
