  Prompt tuning (PT) is an effective approach to adapting pre-trained language
models to downstream tasks. Without a good initialization, prompt tuning
doesn't perform well under few-shot settings. So pre-trained prompt tuning
(PPT) is proposed to initialize prompts by leveraging pre-training data. We
propose MetaPT (Meta-learned Prompt Tuning) to further improve PPT's
initialization by considering latent structure within the pre-training data.
Specifically, we introduce the structure by first clustering pre-training data
into different auxiliary tasks with unsupervised methods. Then we use these
tasks to pre-train prompts with a meta-learning algorithm. Such a process can
make prompts learn a better initialization by discovering commonalities among
these auxiliary tasks. We evaluate our method on seven downstream tasks. Our
MetaPT achieves better and more stable performance than the state-of-the-art
method.
