  Soft prompt tuning is a widely studied parameter-efficient fine-tuning
method. However, it has a clear drawback: many soft tokens must be inserted
into the input sequences to guarantee downstream performance. As a result, soft
prompt tuning is less considered than Low-rank adaptation (LoRA) in the large
language modeling (LLM) era. In this work, we propose a novel prompt tuning
method, Instruction-Aware Prompt Tuning (IAPT), that requires only four soft
tokens. First, we install a parameter-efficient soft prompt generator at each
Transformer layer to generate idiosyncratic soft prompts for each input
instruction. The generated soft prompts can be seen as a semantic summary of
the input instructions and can effectively guide the output generation. Second,
the soft prompt generators are modules with a bottleneck architecture
consisting of a self-attention pooling operation, two linear projections, and
an activation function. Pilot experiments show that prompt generators at
different Transformer layers require different activation functions. Thus, we
propose to learn the idiosyncratic activation functions for prompt generators
automatically with the help of rational functions. We have conducted
experiments on various tasks, and the experimental results demonstrate that (a)
our IAPT method can outperform the recent baselines with comparable tunable
parameters. (b) Our IAPT method is more efficient than LoRA under the
single-backbone multi-tenant setting.
