  The objective of this work is to explore how to effectively and efficiently
adapt pre-trained visual foundation models to various downstream tasks of
semantic segmentation. Previous methods usually fine-tuned the entire networks
for each specific dataset, which will be burdensome to store massive parameters
of these networks. A few recent works attempted to insert some extra trainable
parameters into the frozen networks to learn visual prompts for
parameter-efficient tuning. However, these works showed poor generality as they
were designed specifically for Transformers. Moreover, using limited
information in these schemes, they exhibited a poor capacity to learn
beneficial prompts. To alleviate these issues, we propose a novel Stage-wise
Prompt-Matched Framework for generic and effective visual prompt tuning.
Specifically, to ensure generality, we divide the pre-trained backbone with
frozen parameters into multiple stages and perform prompt learning between
different stages, which makes the proposed scheme applicable to various
architectures of CNN and Transformer. For effective tuning, a lightweight
Semantic-aware Prompt Matcher (SPM) is designed to progressively learn
reasonable prompts with a recurrent mechanism, guided by the rich information
of interim semantic maps. Working as deep matched filter of representation
learning, the proposed SPM can well transform the output of the previous stage
into a desirable input for the next stage, thus achieving the better
matching/stimulating for the pre-trained knowledge. Extensive experiments on
four benchmarks demonstrate that the proposed scheme can achieve a promising
trade-off between parameter efficiency and performance effectiveness. Our code
and models will be released.
