  Prompt tuning is a promising method to fine-tune a pre-trained language model
without retraining its large-scale parameters. Instead, it attaches a soft
prompt to the input text, whereby downstream tasks can be well adapted by
merely learning the embeddings of prompt tokens. Nevertheless, existing methods
still suffer from two challenges: (i) they are hard to balance accuracy and
efficiency. A longer (shorter) soft prompt generally leads to a better(worse)
accuracy but at the cost of more (less) training time. (ii)The performance may
not be consistent when adapting to different downstream tasks. We attribute it
to the same embedding space but responsible for different requirements of
downstream tasks. To address these issues, we propose an Efficient Prompt
Tuning method (EPT) by multi-space projection and prompt fusion. Specifically,
it decomposes a given soft prompt into a shorter prompt and two low-rank
matrices, significantly reducing the training time. Accuracy is also enhanced
by leveraging low-rank matrices and the short prompt as additional knowledge
sources to enrich the semantics of the original short prompt. In addition, we
project the soft prompt into multiple subspaces to improve the performance
consistency, and then adaptively learn the combination weights of different
spaces through a gating network. Experiments on 13 natural language processing
downstream tasks show that our method significantly and consistently
outperforms 11 comparison methods with the relative percentage of improvements
up to 12.9%, and training time decreased by 14%.
