  We investigate a framework for machine reading, inspired by real world
information-seeking problems, where a meta question answering system interacts
with a black box environment. The environment encapsulates a competitive
machine reader based on BERT, providing candidate answers to questions, and
possibly some context. To validate the realism of our formulation, we ask
humans to play the role of a meta-answerer. With just a small snippet of text
around an answer, humans can outperform the machine reader, improving recall.
Similarly, a simple machine meta-answerer outperforms the environment,
improving both precision and recall on the Natural Questions dataset. The
system relies on joint training of answer scoring and the selection of
conditioning information.
