  We present an approach to efficiently and effectively adapt a masked image
modeling (MIM) pre-trained vanilla Vision Transformer (ViT) for object
detection, which is based on our two novel observations: (i) A MIM pre-trained
vanilla ViT encoder can work surprisingly well in the challenging object-level
recognition scenario even with randomly sampled partial observations, e.g.,
only 25% $\sim$ 50% of the input embeddings. (ii) In order to construct
multi-scale representations for object detection from single-scale ViT, a
randomly initialized compact convolutional stem supplants the pre-trained large
kernel patchify stem, and its intermediate features can naturally serve as the
higher resolution inputs of a feature pyramid network without further
upsampling or other manipulations. While the pre-trained ViT is only regarded
as the 3$^{rd}$-stage of our detector's backbone instead of the whole feature
extractor. This results in a ConvNet-ViT hybrid feature extractor. The proposed
detector, named MIMDet, enables a MIM pre-trained vanilla ViT to outperform
hierarchical Swin Transformer by 2.5 box AP and 2.6 mask AP on COCO, and
achieves better results compared with the previous best adapted vanilla ViT
detector using a more modest fine-tuning recipe while converging 2.8$\times$
faster. Code and pre-trained models are available at
https://github.com/hustvl/MIMDet.
