  Prompt tuning is one of the successful approaches for parameter-efficient
tuning of pre-trained language models. Despite being arguably the most
parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
it typically performs worse than other efficient tuning methods and is quite
sensitive to hyper-parameters. In this work, we introduce Residual Prompt
Tuning - a simple and efficient method that significantly improves the
performance and stability of prompt tuning. We propose to reparameterize soft
prompt embeddings using a shallow network with a residual connection. Our
experiments show that Residual Prompt Tuning significantly outperforms prompt
tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
improvement over prompt tuning with T5-Base and allows to reduce the prompt
length by 10x without hurting performance. In addition, we show that our
approach is robust to the choice of learning rate and prompt initialization,
and is effective in few-shot settings.
