  While human evaluation remains best practice for accurately judging the
faithfulness of automatically-generated summaries, few solutions exist to
address the increased difficulty and workload when evaluating long-form
summaries. Through a survey of 162 papers on long-form summarization, we first
shed light on current human evaluation practices surrounding long-form
summaries. We find that 73% of these papers do not perform any human evaluation
on model-generated summaries, while other works face new difficulties that
manifest when dealing with long documents (e.g., low inter-annotator
agreement). Motivated by our survey, we present LongEval, a set of guidelines
for human evaluation of faithfulness in long-form summaries that addresses the
following challenges: (1) How can we achieve high inter-annotator agreement on
faithfulness scores? (2) How can we minimize annotator workload while
maintaining accurate faithfulness scores? and (3) Do humans benefit from
automated alignment between summary and source snippets? We deploy LongEval in
annotation studies on two long-form summarization datasets in different domains
(SQuALITY and PubMed), and we find that switching to a finer granularity of
judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness
scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a
partial annotation of fine-grained units highly correlates with scores from a
full annotation workload (0.89 Kendall's tau using 50% judgments). We release
our human judgments, annotation templates, and our software as a Python library
for future research.
