  Various tasks, such as summarization, multi-hop question answering, or
coreference resolution, are naturally phrased over collections of real-world
documents. Such tasks present a unique set of challenges, revolving around the
lack of coherent narrative structure across documents, which often leads to
contradiction, omission, or repetition of information. Despite their real-world
application and challenging properties, there is currently no benchmark which
specifically measures the abilities of large language models (LLMs) on
multi-document tasks. To bridge this gap, we present SEAM (a Stochastic
Evaluation Approach for Multi-document tasks), a conglomerate benchmark over a
diverse set of multi-document datasets, setting conventional evaluation
criteria, input-output formats, and evaluation protocols. In particular, SEAM
addresses the sensitivity of LLMs to minor prompt variations through repeated
evaluations, where in each evaluation we sample uniformly at random the values
of arbitrary factors (e.g., the order of documents). We evaluate different LLMs
on SEAM finding that multi-document tasks pose a significant challenge for
LLMs, even for state-of-the-art models with 70B parameters. In addition, we
show that the stochastic approach uncovers underlying statistical trends which
cannot be observed in a static benchmark. We hope that SEAM will spur progress
via consistent and meaningful evaluation of multi-document tasks.
