  Transformers have demonstrated great potential in computer vision tasks. To
avoid dense computations of self-attentions in high-resolution visual data,
some recent Transformer models adopt a hierarchical design, where
self-attentions are only computed within local windows. This design
significantly improves the efficiency but lacks global feature reasoning in
early stages. In this work, we design a multi-path structure of the
Transformer, which enables local-to-global reasoning at multiple granularities
in each stage. The proposed framework is computationally efficient and highly
effective. With a marginal increasement in computational overhead, our model
achieves notable improvements in both image classification and semantic
segmentation. Code is available at https://github.com/ljpadam/LG-Transformer
