  Prompt tuning approaches, which learn task-specific soft prompts for a
downstream task conditioning on frozen pre-trained models, have attracted
growing interest due to its parameter efficiency. With large language models
and sufficient training data, prompt tuning performs comparably to full-model
tuning. However, with limited training samples in few-shot settings, prompt
tuning fails to match the performance of full-model fine-tuning. In this work,
we focus on improving the few-shot performance of prompt tuning by transferring
knowledge from soft prompts of source tasks. Recognizing the good
generalization capabilities of ensemble methods in low-data regime, we first
experiment and show that a simple ensemble of model predictions based on
different source prompts, outperforms existing multi-prompt knowledge transfer
approaches such as source prompt fusion in the few-shot setting. Motivated by
this observation, we further investigate model ensembles and propose
Sample-specific Ensemble of Source Models (SESoM). SESoM learns to adjust the
contribution of each source model for each target sample separately when
ensembling source model outputs. Through this way, SESoM inherits the superior
generalization of model ensemble approaches and simultaneously captures the
sample-specific competence of each source prompt. We conduct experiments across
a diverse set of eight NLP tasks using models of different scales (T5-{base,
large, XL}) and find that SESoM consistently outperforms the existing models of
the same as well as larger parametric scale by a large margin.
