  While there has been substantial progress in text comprehension through
simple factoid question answering, more holistic comprehension of a discourse
still presents a major challenge (Dunietz et al., 2020). Someone critically
reflecting on a text as they read it will pose curiosity-driven, often
open-ended questions, which reflect deep understanding of the content and
require complex reasoning to answer (Ko et al., 2020; Westera et al., 2020). A
key challenge in building and evaluating models for this type of discourse
comprehension is the lack of annotated data, especially since collecting
answers to such questions requires high cognitive load for annotators. This
paper presents a novel paradigm that enables scalable data collection targeting
the comprehension of news documents, viewing these questions through the lens
of discourse. The resulting corpus, DCQA (Discourse Comprehension by Question
Answering), captures both discourse and semantic links between sentences in the
form of free-form, open-ended questions. On an evaluation set that we annotated
on questions from Ko et al. (2020), we show that DCQA provides valuable
supervision for answering open-ended questions. We additionally design
pre-training methods utilizing existing question-answering resources, and use
synthetic data to accommodate unanswerable questions.
