  Masked image modeling (MIM) has become a popular strategy for self-supervised
learning~(SSL) of visual representations with Vision Transformers. A
representative MIM model, the masked auto-encoder (MAE), randomly masks a
subset of image patches and reconstructs the masked patches given the unmasked
patches. Concurrently, many recent works in self-supervised learning utilize
the student/teacher paradigm which provides the student with an additional
target based on the output of a teacher composed of an exponential moving
average (EMA) of previous students. Although common, relatively little is known
about the dynamics of the interaction between the student and teacher. Through
analysis on a simple linear model, we find that the teacher conditionally
removes previous gradient directions based on feature similarities which
effectively acts as a conditional momentum regularizer. From this analysis, we
present a simple SSL method, the Reconstruction-Consistent Masked Auto-Encoder
(RC-MAE) by adding an EMA teacher to MAE. We find that RC-MAE converges faster
and requires less memory usage than state-of-the-art self-distillation methods
during pre-training, which may provide a way to enhance the practicality of
prohibitively expensive self-supervised learning of Vision Transformer models.
Additionally, we show that RC-MAE achieves more robustness and better
performance compared to MAE on downstream tasks such as ImageNet-1K
classification, object detection, and instance segmentation.
