  Plug-and-play functionality allows deep learning models to adapt well to
different tasks without requiring any parameters modified. Recently,
prefix-tuning was shown to be a plug-and-play method on various text generation
tasks by simply inserting corresponding continuous vectors into the inputs.
However, sequence labeling tasks invalidate existing plug-and-play methods
since different label sets demand changes to the architecture of the model
classifier. In this work, we propose the use of label word prediction instead
of classification to totally reuse the architecture of pre-trained models for
sequence labeling tasks. Specifically, for each task, a label word set is first
constructed by selecting a high-frequency word for each class respectively, and
then, task-specific vectors are inserted into the inputs and optimized to
manipulate the model predictions towards the corresponding label words. As a
result, by simply switching the plugin vectors on the input, a frozen
pre-trained language model is allowed to perform different tasks. Experimental
results on three sequence labeling tasks show that the performance of the
proposed method can achieve comparable performance with standard fine-tuning
with only 0.1\% task-specific parameters. In addition, our method is up to 70
times faster than non-plug-and-play methods while switching different tasks
under the resource-constrained scenario.
