  Parameter-efficient fine-tuning (PEFT) has emerged as a popular approach for
adapting pre-trained Vision Transformer (ViT) models to downstream
applications. While current PEFT methods achieve parameter efficiency, they
overlook GPU memory and time efficiency during both fine-tuning and inference,
due to the repeated computation of redundant tokens in the ViT architecture.
This falls short of practical requirements for downstream task adaptation. In
this paper, we propose \textbf{Sparse-Tuning}, a novel tuning paradigm that
substantially enhances both fine-tuning and inference efficiency for
pre-trained ViT models. Sparse-Tuning efficiently fine-tunes the pre-trained
ViT by sparsely preserving the informative tokens and merging redundant ones,
enabling the ViT to focus on the foreground while reducing computational costs
on background regions in the images. To accurately distinguish informative
tokens from uninformative ones, we introduce a tailored Dense Adapter, which
establishes dense connections across different encoder layers in the ViT,
thereby enhancing the representational capacity and quality of token
sparsification. Empirical results on VTAB-1K, three complete image datasets,
and two complete video datasets demonstrate that Sparse-Tuning reduces the
GFLOPs to \textbf{62\%-70\%} of the original ViT-B while achieving
state-of-the-art performance. Source code is available at
\url{https://github.com/liuting20/Sparse-Tuning}.
