  The ability to derive useful information by asking clarifying questions (ACQ)
is an important element of real life collaboration on reasoning tasks, such as
question answering (QA). Existing natural language ACQ challenges, however,
evaluate generations based on word overlap rather than the value of the
information itself. Word overlap is often an inappropriate metric for question
generation since many different questions could be useful in a given situation,
and a single question can be phrased many different ways. Instead, we propose
evaluating questions pragmatically based on the value of the information they
retrieve. Here we present a definition and framework for natural language
pragmatic asking of clarifying questions (PACQ), the problem of generating
questions that result in answers useful for a reasoning task. We also present
fact-level masking (FLM), a procedure for converting natural language datasets
into self-supervised PACQ datasets by omitting particular critical facts.
Finally, we generate a PACQ dataset from the HotpotQA dataset using FLM and
evaluate several zero-shot language models on it. Our experiments show that
current zero-shot models struggle to ask questions that retrieve useful
information, as compared to human annotators. These results demonstrate an
opportunity to use FLM datasets and the PACQ framework to objectively evaluate
and improve question generation and other language models.
