  Parameter-efficient transfer learning (PETL) based on large-scale pre-trained
foundation models has achieved great success in various downstream
applications. Existing tuning methods, such as prompt, prefix, and adapter,
perform task-specific lightweight adjustments to different parts of the
original architecture. However, they take effect on only some parts of the
pre-trained models, i.e., only the feed-forward layers or the self-attention
layers, which leaves the remaining frozen structures unable to adapt to the
data distributions of downstream tasks. Further, the existing structures are
strongly coupled with the Transformers, hindering parameter-efficient
deployment as well as the design flexibility for new approaches. In this paper,
we revisit the design paradigm of PETL and derive a unified framework U-Tuning
for parameter-efficient transfer learning, which is composed of an operation
with frozen parameters and a unified tuner that adapts the operation for
downstream applications. The U-Tuning framework can simultaneously encompass
existing methods and derive new approaches for parameter-efficient transfer
learning, which prove to achieve on-par or better performances on CIFAR-100 and
FGVC datasets when compared with existing PETL methods.
