  While fine-tuning of pre-trained language models generally helps to overcome
the lack of labelled training samples, it also displays model performance
instability. This instability mainly originates from randomness in
initialisation or data shuffling. To address this, researchers either modify
the training process or augment the available samples, which typically results
in increased computational costs. We propose a new mitigation strategy, called
Delayed Ensemble with Noisy Interpolation (DENI), that leverages the strengths
of ensembling, noise regularisation and model interpolation, while retaining
computational efficiency. We compare DENI with 9 representative mitigation
strategies across 3 models, 4 tuning strategies and 7 text classification
datasets. We show that: 1) DENI outperforms the best performing mitigation
strategy (Ensemble), while using only a fraction of its cost; 2) the mitigation
strategies are beneficial for parameter-efficient fine-tuning (PEFT) methods,
outperforming full fine-tuning in specific cases; and 3) combining DENI with
data augmentation often leads to even more effective instability mitigation.
