  We present a pipeline of Image to Vector (Img2Vec) for masked image modeling
(MIM) with deep features. To study which type of deep features is appropriate
for MIM as a learning target, we propose a simple MIM framework with serials of
well-trained self-supervised models to convert an Image to a feature Vector as
the learning target of MIM, where the feature extractor is also known as a
teacher model. Surprisingly, we empirically find that an MIM model benefits
more from image features generated by some lighter models (e.g., ResNet-50,
26M) than from those by a cumbersome teacher like Transformer-based models
(e.g., ViT-Large, 307M). To analyze this remarkable phenomenon, we devise a
novel attribute, token diversity, to evaluate the characteristics of generated
features from different models. Token diversity measures the feature
dissimilarity among different tokens. Through extensive experiments and
visualizations, we hypothesize that beyond the acknowledgment that a large
model can improve MIM, a high token-diversity of a teacher model is also
crucial. Based on the above discussion, Img2Vec adopts a teacher model with
high token-diversity to generate image features. Img2Vec pre-trained on
ImageNet unlabeled data with ViT-B yields 85.1\% top-1 accuracy on fine-tuning.
Moreover, we scale up Img2Vec on larger models, ViT-L and ViT-H, and get
$86.7\%$ and $87.5\%$ accuracy respectively. It also achieves state-of-the-art
results on other downstream tasks, e.g., 51.8\% mAP on COCO and 50.7\% mIoU on
ADE20K. Img2Vec is a simple yet effective framework tailored to deep feature
MIM learning, accomplishing superb comprehensive performance on representative
vision tasks.
