  Commonly adopted metrics for extractive summarization focus on lexical
overlap at the token level. In this paper, we present a facet-aware evaluation
setup for better assessment of the information coverage in extracted summaries.
Specifically, we treat each sentence in the reference summary as a
\textit{facet}, identify the sentences in the document that express the
semantics of each facet as \textit{support sentences} of the facet, and
automatically evaluate extractive summarization methods by comparing the
indices of extracted sentences and support sentences of all the facets in the
reference summary. To facilitate this new evaluation setup, we construct an
extractive version of the CNN/Daily Mail dataset and perform a thorough
quantitative investigation, through which we demonstrate that facet-aware
evaluation manifests better correlation with human judgment than ROUGE, enables
fine-grained evaluation as well as comparative analysis, and reveals valuable
insights of state-of-the-art summarization methods. Data can be found at
https://github.com/morningmoni/FAR.
