  State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain
Question Answering (QA) are typically trained for span selection using
distantly supervised positive examples and heuristically retrieved negative
examples. This training scheme possibly explains empirical observations that
these models achieve a high recall amongst their top few predictions, but a low
overall accuracy, motivating the need for answer re-ranking. We develop a
simple and effective re-ranking approach (RECONSIDER) for span-extraction
tasks, that improves upon the performance of large pre-trained MRC models.
RECONSIDER is trained on positive and negative examples extracted from high
confidence predictions of MRC models, and uses in-passage span annotations to
perform span-focused re-ranking over a smaller candidate set. As a result,
RECONSIDER learns to eliminate close false positive passages, and achieves a
new state of the art on four QA tasks, including 45.5% Exact Match accuracy on
Natural Questions with real user questions, and 61.7% on TriviaQA.
