  In computer vision, it has achieved great transfer learning performance via
adapting large-scale pretrained vision models (e.g., vision transformers) to
downstream tasks. Common approaches for model adaptation either update all
model parameters or leverage linear probes. In this paper, we aim to study
parameter-efficient model adaptation strategies for vision transformers on the
image classification task. We formulate efficient model adaptation as a
subspace training problem and perform a comprehensive benchmarking over
different efficient adaptation methods. We conduct an empirical study on each
efficient model adaptation method focusing on its performance alongside
parameter cost. Furthermore, we propose a parameter-efficient model adaptation
framework, which first selects submodules by measuring local intrinsic
dimensions and then projects them into subspace for further decomposition via a
novel Kronecker Adaptation (KAdaptation) method. We analyze and compare our
method with a diverse set of baseline model adaptation methods (including
state-of-the-art methods for pretrained language models). Our method performs
the best in terms of the tradeoff between accuracy and parameter efficiency
across 20 image classification datasets under the few-shot setting and 7 image
classification datasets under the full-shot setting.
