  Open-domain question answering (QA) is an important problem in AI and NLP
that is emerging as a bellwether for progress on the generalizability of AI
methods and techniques. Much of the progress in open-domain QA systems has been
realized through advances in information retrieval methods and corpus
construction. In this paper, we focus on the recently introduced ARC Challenge
dataset, which contains 2,590 multiple choice questions authored for
grade-school science exams. These questions are selected to be the most
challenging for current QA systems, and current state of the art performance is
only slightly better than random chance. We present a system that rewrites a
given question into queries that are used to retrieve supporting text from a
large corpus of science-related text. Our rewriter is able to incorporate
background knowledge from ConceptNet and -- in tandem with a generic textual
entailment system trained on SciTail that identifies support in the retrieved
results -- outperforms several strong baselines on the end-to-end QA task
despite only being trained to identify essential terms in the original source
question. We use a generalizable decision methodology over the retrieved
evidence and answer candidates to select the best answer. By combining query
rewriting, background knowledge, and textual entailment our system is able to
outperform several strong baselines on the ARC dataset.
