  The quality of a summarization evaluation metric is quantified by calculating
the correlation between its scores and human annotations across a large number
of summaries. Currently, it is unclear how precise these correlation estimates
are, nor whether differences between two metrics' correlations reflect a true
difference or if it is due to mere chance. In this work, we address these two
problems by proposing methods for calculating confidence intervals and running
hypothesis tests for correlations using two resampling methods, bootstrapping
and permutation. After evaluating which of the proposed methods is most
appropriate for summarization through two simulation experiments, we analyze
the results of applying these methods to several different automatic evaluation
metrics across three sets of human annotations. We find that the confidence
intervals are rather wide, demonstrating high uncertainty in the reliability of
automatic metrics. Further, although many metrics fail to show statistical
improvements over ROUGE, two recent works, QAEval and BERTScore, do in some
evaluation settings.
