  Humans observe and interact with the world to acquire knowledge. However,
most existing machine reading comprehension (MRC) tasks miss the interactive,
information-seeking component of comprehension. Such tasks present models with
static documents that contain all necessary information, usually concentrated
in a single short substring. Thus, models can achieve strong performance
through simple word- and phrase-based pattern matching. We address this problem
by formulating a novel text-based question answering task: Question Answering
with Interactive Text (QAit). In QAit, an agent must interact with a partially
observable text-based environment to gather information required to answer
questions. QAit poses questions about the existence, location, and attributes
of objects found in the environment. The data is built using a text-based game
generator that defines the underlying dynamics of interaction with the
environment. We propose and evaluate a set of baseline models for the QAit task
that includes deep reinforcement learning agents. Experiments show that the
task presents a major challenge for machine reading systems, while humans solve
it with relative ease.
