  Prompt learning is an efficient approach to adapt transformers by inserting
learnable set of parameters into the input and intermediate representations of
a pre-trained model. In this work, we present Expressive Prompts with Residuals
(EXPRES) which modifies the prompt learning paradigm specifically for effective
adaptation of vision transformers (ViT). Out method constructs downstream
representations via learnable ``output'' tokens, that are akin to the learned
class tokens of the ViT. Further for better steering of the downstream
representation processed by the frozen transformer, we introduce residual
learnable tokens that are added to the output of various computations. We apply
EXPRES for image classification, few shot learning, and semantic segmentation,
and show our method is capable of achieving state of the art prompt tuning on
3/3 categories of the VTAB benchmark. In addition to strong performance, we
observe that our approach is an order of magnitude more prompt efficient than
existing visual prompting baselines. We analytically show the computational
benefits of our approach over weight space adaptation techniques like
finetuning. Lastly we systematically corroborate the architectural design of
our method via a series of ablation experiments.
