  Recently, masked image modeling (MIM) has offered a new methodology of
self-supervised pre-training of vision transformers. A key idea of efficient
implementation is to discard the masked image patches (or tokens) throughout
the target network (encoder), which requires the encoder to be a plain vision
transformer (e.g., ViT), albeit hierarchical vision transformers (e.g., Swin
Transformer) have potentially better properties in formulating vision inputs.
In this paper, we offer a new design of hierarchical vision transformers named
HiViT (short for Hierarchical ViT) that enjoys both high efficiency and good
performance in MIM. The key is to remove the unnecessary "local inter-unit
operations", deriving structurally simple hierarchical vision transformers in
which mask-units can be serialized like plain vision transformers. For this
purpose, we start with Swin Transformer and (i) set the masking unit size to be
the token size in the main stage of Swin Transformer, (ii) switch off
inter-unit self-attentions before the main stage, and (iii) eliminate all
operations after the main stage. Empirical studies demonstrate the advantageous
performance of HiViT in terms of fully-supervised, self-supervised, and
transfer learning. In particular, in running MAE on ImageNet-1K, HiViT-B
reports a +0.6% accuracy gain over ViT-B and a 1.9$\times$ speed-up over
Swin-B, and the performance gain generalizes to downstream tasks of detection
and segmentation. Code will be made publicly available.
