  Prompt tuning, in which a base pretrained model is adapted to each task via
conditioning on learned prompt vectors, has emerged as a promising approach for
efficiently adapting large language models to multiple downstream tasks.
However, existing methods typically learn soft prompt vectors from scratch, and
it has not been clear how to exploit the rich cross-task knowledge with prompt
vectors in a multitask learning setting. We propose multitask prompt tuning
(MPT), which first learns a single transferable prompt by distilling knowledge
from multiple task-specific source prompts. We then learn multiplicative low
rank updates to this shared prompt to efficiently adapt it to each downstream
target task. Extensive experiments on 23 NLP datasets demonstrate that our
proposed approach outperforms the state-of-the-art methods, including the full
finetuning baseline in some cases, despite only tuning 0.035% as many
task-specific parameters.
