  Prompt tuning is a modular and efficient solution for training large language
models (LLMs). One of its main advantages is task modularity, making it
suitable for multi-task problems. However, current soft-prompt-based methods
often sacrifice multi-task modularity, requiring the training process to be
fully or partially repeated for each newly added task. While recent work on
task vectors applied arithmetic operations on full model weights to achieve the
desired multi-task performance, a similar approach for soft-prompts is still
missing. To this end, we introduce Task Prompt Vectors, created by element-wise
difference between weights of tuned soft-prompts and their random
initialization. Experimental results on 12 NLU datasets show that task prompt
vectors can be used in low-resource settings to effectively initialize prompt
tuning on similar tasks. In addition, we show that task prompt vectors are
independent of the random initialization of prompt tuning. This allows prompt
arithmetics with the pre-trained vectors from different tasks. In this way, by
arithmetic addition of task prompt vectors from multiple tasks, we are able to
outperform a state-of-the-art baseline in some cases.
