  The invention of transformer-based models such as BERT, GPT, and RoBERTa has
enabled researchers and financial companies to finetune these powerful models
and use them in different downstream tasks to achieve state-of-the-art
performance. Recently, a lightweight alternative (approximately 0.1% - 3% of
the original model parameters) to fine-tuning, known as prefix tuning has been
introduced. This method freezes the model parameters and only updates the
prefix to achieve performance comparable to full fine-tuning. Prefix tuning
enables researchers and financial practitioners to achieve similar results with
much fewer parameters. In this paper, we explore the robustness of prefix
tuning when facing noisy data. Our experiments demonstrate that fine-tuning is
more robust to noise than prefix tuning -- the latter method faces a
significant decrease in performance on most corrupted data sets with increasing
noise levels. Furthermore, prefix tuning has high variances in the F1 scores
compared to fine-tuning in many corruption methods. We strongly advocate that
caution should be carefully taken when applying the state-of-the-art prefix
tuning method to noisy data.
