  The formidable accomplishment of Transformers in natural language processing
has motivated the researchers in the computer vision community to build Vision
Transformers. Compared with the Convolution Neural Networks (CNN), a Vision
Transformer has a larger receptive field which is capable of characterizing the
long-range dependencies. Nevertheless, the large receptive field of Vision
Transformer is accompanied by the huge computational cost. To boost efficiency,
the window-based Vision Transformers emerge. They crop an image into several
local windows, and the self-attention is conducted within each window. To bring
back the global receptive field, window-based Vision Transformers have devoted
a lot of efforts to achieving cross-window communications by developing several
sophisticated operations. In this work, we check the necessity of the key
design element of Swin Transformer, the shifted window partitioning. We
discover that a simple depthwise convolution is sufficient for achieving
effective cross-window communications. Specifically, with the existence of the
depthwise convolution, the shifted window configuration in Swin Transformer
cannot lead to an additional performance improvement. Thus, we degenerate the
Swin Transformer to a plain Window-based (Win) Transformer by discarding
sophisticated shifted window partitioning. The proposed Win Transformer is
conceptually simpler and easier for implementation than Swin Transformer.
Meanwhile, our Win Transformer achieves consistently superior performance than
Swin Transformer on multiple computer vision tasks, including image
recognition, semantic segmentation, and object detection.
