  Masked autoencoders have become popular training paradigms for
self-supervised visual representation learning. These models randomly mask a
portion of the input and reconstruct the masked portion according to the target
representations. In this paper, we first show that a careful choice of the
target representation is unnecessary for learning good representations, since
different targets tend to derive similarly behaved models. Driven by this
observation, we propose a multi-stage masked distillation pipeline and use a
randomly initialized model as the teacher, enabling us to effectively train
high-capacity models without any efforts to carefully design target
representations. Interestingly, we further explore using teachers of larger
capacity, obtaining distilled students with remarkable transferring ability. On
different tasks of classification, transfer learning, object detection, and
semantic segmentation, the proposed method to perform masked knowledge
distillation with bootstrapped teachers (dBOT) outperforms previous
self-supervised methods by nontrivial margins. We hope our findings, as well as
the proposed method, could motivate people to rethink the roles of target
representations in pre-training masked autoencoders.The code and pre-trained
models are publicly available at https://github.com/liuxingbin/dbot.
