  The scarcity of comprehensive up-to-date studies on evaluation metrics for
text summarization and the lack of consensus regarding evaluation protocols
continue to inhibit progress. We address the existing shortcomings of
summarization evaluation methods along five dimensions: 1) we re-evaluate 14
automatic evaluation metrics in a comprehensive and consistent fashion using
neural summarization model outputs along with expert and crowd-sourced human
annotations, 2) we consistently benchmark 23 recent summarization models using
the aforementioned automatic evaluation metrics, 3) we assemble the largest
collection of summaries generated by models trained on the CNN/DailyMail news
dataset and share it in a unified format, 4) we implement and share a toolkit
that provides an extensible and unified API for evaluating summarization models
across a broad range of automatic metrics, 5) we assemble and share the largest
and most diverse, in terms of model types, collection of human judgments of
model-generated summaries on the CNN/Daily Mail dataset annotated by both
expert judges and crowd-source workers. We hope that this work will help
promote a more complete evaluation protocol for text summarization as well as
advance research in developing evaluation metrics that better correlate with
human judgments.
