  This paper studies the potential of distilling knowledge from pre-trained
models, especially Masked Autoencoders. Our approach is simple: in addition to
optimizing the pixel reconstruction loss on masked inputs, we minimize the
distance between the intermediate feature map of the teacher model and that of
the student model. This design leads to a computationally efficient knowledge
distillation framework, given 1) only a small visible subset of patches is
used, and 2) the (cumbersome) teacher model only needs to be partially
executed, ie, forward propagate inputs through the first few layers, for
obtaining intermediate feature maps. Compared to directly distilling fine-tuned
models, distilling pre-trained models substantially improves downstream
performance. For example, by distilling the knowledge from an MAE pre-trained
ViT-L into a ViT-B, our method achieves 84.0% ImageNet top-1 accuracy,
outperforming the baseline of directly distilling a fine-tuned ViT-L by 1.2%.
More intriguingly, our method can robustly distill knowledge from teacher
models even with extremely high masking ratios: e.g., with 95% masking ratio
where merely TEN patches are visible during distillation, our ViT-B
competitively attains a top-1 ImageNet accuracy of 83.6%; surprisingly, it can
still secure 82.4% top-1 ImageNet accuracy by aggressively training with just
FOUR visible patches (98% masking ratio). The code and models are publicly
available at https://github.com/UCSC-VLAA/DMAE.
