  The success of large-scale pre-trained models has established fine-tuning as
a standard method for achieving significant improvements in downstream tasks.
However, fine-tuning the entire parameter set of a pre-trained model is costly.
Parameter-efficient transfer learning (PETL) has recently emerged as a
cost-effective alternative for adapting pre-trained models to downstream tasks.
Despite its advantages, the increasing model size and input resolution present
challenges for PETL, as the training memory consumption is not reduced as
effectively as the parameter usage. In this paper, we introduce Fine-grained
Prompt Tuning plus (FPT+), a PETL method designed for high-resolution medical
image classification, which significantly reduces memory consumption compared
to other PETL methods. FPT+ performs transfer learning by training a
lightweight side network and accessing pre-trained knowledge from a large
pre-trained model (LPM) through fine-grained prompts and fusion modules.
Specifically, we freeze the LPM and construct a learnable lightweight side
network. The frozen LPM processes high-resolution images to extract
fine-grained features, while the side network employs the corresponding
down-sampled low-resolution images to minimize the memory usage. To enable the
side network to leverage pre-trained knowledge, we propose fine-grained prompts
and fusion modules, which collaborate to summarize information through the
LPM's intermediate activations. We evaluate FPT+ on eight medical image
datasets of varying sizes, modalities, and complexities. Experimental results
demonstrate that FPT+ outperforms other PETL methods, using only 1.03% of the
learnable parameters and 3.18% of the memory required for fine-tuning an entire
ViT-B model. Our code is available at https://github.com/YijinHuang/FPT.
