  Automatic summarization with pre-trained language models has led to
impressively fluent results, but is prone to 'hallucinations', low performance
on non-news genres, and outputs which are not exactly summaries. Targeting ACL
2023's 'Reality Check' theme, we present GUMSum, a small but carefully crafted
dataset of English summaries in 12 written and spoken genres for evaluation of
abstractive summarization. Summaries are highly constrained, focusing on
substitutive potential, factuality, and faithfulness. We present guidelines and
evaluate human agreement as well as subjective judgments on recent system
outputs, comparing general-domain untuned approaches, a fine-tuned one, and a
prompt-based approach, to human performance. Results show that while GPT3
achieves impressive scores, it still underperforms humans, with varying quality
across genres. Human judgments reveal different types of errors in supervised,
prompted, and human-generated summaries, shedding light on the challenges of
producing a good summary.
