  Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in
adapting the pre-trained language models to downstream tasks while only
updating a small number of parameters. Despite the success, most existing
methods independently adapt to each task without considering knowledge transfer
between tasks and are limited to low-data regimes. To overcome this issue, we
propose Prototype-based HyperAdapter (PHA), a novel framework built on the
adapter-tuning and hypernetwork. It introduces an instance-dense retriever and
a prototypical hypernetwork to generate the conditional modules in a
sample-efficient manner. This leads to comparable performance improvements
against existing PEFT methods on multi-task learning and few-shot transfer
learning. More importantly, when the available data size gets smaller, our
method outperforms other strong baselines by a large margin. Based on our
extensive empirical experiments across various datasets, we demonstrate that
PHA strikes a better trade-off between trainable parameters, accuracy on stream
tasks, and sample efficiency.
