  Fine-tuning pre-trained language models on downstream tasks with varying
random seeds has been shown to be unstable, especially on small datasets. Many
previous studies have investigated this instability and proposed methods to
mitigate it. However, most studies only used the standard deviation of
performance scores (SD) as their measure, which is a narrow characterization of
instability. In this paper, we analyze SD and six other measures quantifying
instability at different levels of granularity. Moreover, we propose a
systematic framework to evaluate the validity of these measures. Finally, we
analyze the consistency and difference between different measures by
reassessing existing instability mitigation methods. We hope our results will
inform the development of better measurements of fine-tuning instability.
