  An abundance of datasets and availability of reliable evaluation metrics have
resulted in strong progress in factoid question answering (QA). This progress,
however, does not easily transfer to the task of long-form QA, where the goal
is to answer questions that require in-depth explanations. The hurdles include
(i) a lack of high-quality data, and (ii) the absence of a well-defined notion
of the answer's quality. In this work, we address these problems by (i)
releasing a novel dataset and a task that we call ASQA (Answer Summaries for
Questions which are Ambiguous); and (ii) proposing a reliable metric for
measuring performance on ASQA. Our task focuses on factoid questions that are
ambiguous, that is, have different correct answers depending on interpretation.
Answers to ambiguous questions should synthesize factual information from
multiple sources into a long-form summary that resolves the ambiguity. In
contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear
notion of correctness: a user faced with a good summary should be able to
answer different interpretations of the original ambiguous question. We use
this notion of correctness to define an automated metric of performance for
ASQA. Our analysis demonstrates an agreement between this metric and human
judgments, and reveals a considerable gap between human performance and strong
baselines.
