  Prompt tuning is a parameter-efficient approach to adapting pre-trained
language models to downstream tasks. Although prompt tuning has been shown to
match the performance of full model tuning when training data is sufficient, it
tends to struggle in few-shot learning settings. In this paper, we present
Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot
learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks.
On downstream tasks, the pre-trained prompts are selectively activated and
combined, leading to strong compositional generalization to unseen tasks. To
bridge the gap between pre-training and fine-tuning, we formulate upstream and
downstream tasks into a unified machine reading comprehension task. Extensive
experiments under two learning paradigms, i.e., gradient descent and black-box
tuning, show that MP2 significantly outperforms prompt tuning, full model
tuning, and prior prompt pre-training methods in few-shot settings. In
addition, we demonstrate that MP2 can achieve surprisingly fast and strong
adaptation to downstream tasks by merely learning 8 parameters to combine the
pre-trained modular prompts.
