  In recent years, Transformers have achieved remarkable progress in computer
vision tasks. However, their global modeling often comes with substantial
computational overhead, in stark contrast to the human eye's efficient
information processing. Inspired by the human eye's sparse scanning mechanism,
we propose a \textbf{S}parse \textbf{S}can \textbf{S}elf-\textbf{A}ttention
mechanism ($\rm{S}^3\rm{A}$). This mechanism predefines a series of Anchors of
Interest for each token and employs local attention to efficiently model the
spatial information around these anchors, avoiding redundant global modeling
and excessive focus on local information. This approach mirrors the human eye's
functionality and significantly reduces the computational load of vision
models. Building on $\rm{S}^3\rm{A}$, we introduce the \textbf{S}parse
\textbf{S}can \textbf{Vi}sion \textbf{T}ransformer (SSViT). Extensive
experiments demonstrate the outstanding performance of SSViT across a variety
of tasks. Specifically, on ImageNet classification, without additional
supervision or training data, SSViT achieves top-1 accuracies of
\textbf{84.4\%/85.7\%} with \textbf{4.4G/18.2G} FLOPs. SSViT also excels in
downstream tasks such as object detection, instance segmentation, and semantic
segmentation. Its robustness is further validated across diverse datasets. Code
will be available at \url{https://github.com/qhfan/SSViT}.
