  While numerous architectures for long-range language models (LRLMs) have
recently been proposed, a meaningful evaluation of their discourse-level
language understanding capabilities has not yet followed. To this end, we
introduce ChapterBreak, a challenge dataset that provides an LRLM with a long
segment from a narrative that ends at a chapter boundary and asks it to
distinguish the beginning of the ground-truth next chapter from a set of
negative segments sampled from the same narrative. A fine-grained human
annotation reveals that our dataset contains many complex types of chapter
transitions (e.g., parallel narratives, cliffhanger endings) that require
processing global context to comprehend. Experiments on ChapterBreak show that
existing LRLMs fail to effectively leverage long-range context, substantially
underperforming a segment-level model trained directly for this task. We
publicly release our ChapterBreak dataset to spur more principled future
research into LRLMs.
