  Summarization datasets are often assembled either by scraping naturally
occurring public-domain summaries -- which are nearly always in
difficult-to-work-with technical domains -- or by using approximate heuristics
to extract them from everyday text -- which frequently yields unfaithful
summaries. In this work, we turn to a slower but more straightforward approach
to developing summarization benchmark data: We hire highly-qualified
contractors to read stories and write original summaries from scratch. To
amortize reading time, we collect five summaries per document, with the first
giving an overview and the subsequent four addressing specific questions. We
use this protocol to collect SQuALITY, a dataset of question-focused summaries
built on the same public-domain short stories as the multiple-choice dataset
QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization
systems show that our dataset is challenging and that existing automatic
evaluation metrics are weak indicators of quality.
