  Human evaluation is the foundation upon which the evaluation of both
summarization systems and automatic metrics rests. However, existing human
evaluation studies for summarization either exhibit a low inter-annotator
agreement or have insufficient scale, and an in-depth analysis of human
evaluation is lacking. Therefore, we address the shortcomings of existing
summarization evaluation along the following axes: (1) We propose a modified
summarization salience protocol, Atomic Content Units (ACUs), which is based on
fine-grained semantic units and allows for a high inter-annotator agreement.
(2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large
human evaluation dataset consisting of 22,000 summary-level annotations over 28
top-performing systems on three datasets. (3) We conduct a comparative study of
four human evaluation protocols, underscoring potential confounding factors in
evaluation setups. (4) We evaluate 50 automatic metrics and their variants
using the collected human annotations across evaluation protocols and
demonstrate how our benchmark leads to more statistically stable and
significant results. The metrics we benchmarked include recent methods based on
large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings
have important implications for evaluating LLMs, as we show that LLMs adjusted
by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation,
which is affected by the annotators' prior, input-agnostic preferences, calling
for more robust, targeted evaluation methods.
