  The topic of summarization evaluation has recently attracted a surge of
attention due to the rapid development of abstractive summarization systems.
However, the formulation of the task is rather ambiguous, neither the
linguistic nor the natural language processing community has succeeded in
giving a mutually agreed-upon definition. Due to this lack of well-defined
formulation, a large number of popular abstractive summarization datasets are
constructed in a manner that neither guarantees validity nor meets one of the
most essential criteria of summarization: factual consistency. In this paper,
we address this issue by combining state-of-the-art factual consistency models
to identify the problematic instances present in popular summarization
datasets. We release SummFC, a filtered summarization dataset with improved
factual consistency, and demonstrate that models trained on this dataset
achieve improved performance in nearly all quality aspects. We argue that our
dataset should become a valid benchmark for developing and evaluating
summarization systems.
