  Prior works have proposed several strategies to reduce the computational cost
of self-attention mechanism. Many of these works consider decomposing the
self-attention procedure into regional and local feature extraction procedures
that each incurs a much smaller computational complexity. However, regional
information is typically only achieved at the expense of undesirable
information lost owing to down-sampling. In this paper, we propose a novel
Transformer architecture that aims to mitigate the cost issue, named Dual
Vision Transformer (Dual-ViT). The new architecture incorporates a critical
semantic pathway that can more efficiently compress token vectors into global
semantics with reduced order of complexity. Such compressed global semantics
then serve as useful prior information in learning finer pixel level details,
through another constructed pixel pathway. The semantic pathway and pixel
pathway are then integrated together and are jointly trained, spreading the
enhanced self-attention information in parallel through both of the pathways.
Dual-ViT is henceforth able to reduce the computational complexity without
compromising much accuracy. We empirically demonstrate that Dual-ViT provides
superior accuracy than SOTA Transformer architectures with reduced training
complexity. Source code is available at
\url{https://github.com/YehLi/ImageNetModel}.
