  We present an efficient approach for Masked Image Modeling (MIM) with
hierarchical Vision Transformers (ViTs), allowing the hierarchical ViTs to
discard masked patches and operate only on the visible ones. Our approach
consists of three key designs. First, for window attention, we propose a Group
Window Attention scheme following the Divide-and-Conquer strategy. To mitigate
the quadratic complexity of the self-attention w.r.t. the number of patches,
group attention encourages a uniform partition that visible patches within each
local window of arbitrary size can be grouped with equal size, where masked
self-attention is then performed within each group. Second, we further improve
the grouping strategy via the Dynamic Programming algorithm to minimize the
overall computation cost of the attention on the grouped patches. Third, as for
the convolution layers, we convert them to the Sparse Convolution that works
seamlessly with the sparse data, i.e., the visible patches in MIM. As a result,
MIM can now work on most, if not all, hierarchical ViTs in a green and
efficient way. For example, we can train the hierarchical ViTs, e.g., Swin
Transformer and Twins Transformer, about 2.7$\times$ faster and reduce the GPU
memory usage by 70%, while still enjoying competitive performance on ImageNet
classification and the superiority on downstream COCO object detection
benchmarks. Code and pre-trained models have been made publicly available at
https://github.com/LayneH/GreenMIM.
