  We present an accurate and interpretable method for answer extraction in
machine reading comprehension that is reminiscent of case-based reasoning (CBR)
from classical AI. Our method (CBR-MRC) builds upon the hypothesis that
contextualized answers to similar questions share semantic similarities with
each other. Given a test question, CBR-MRC first retrieves a set of similar
cases from a nonparametric memory and then predicts an answer by selecting the
span in the test context that is most similar to the contextualized
representations of answers in the retrieved cases. The semi-parametric nature
of our approach allows it to attribute a prediction to the specific set of
evidence cases, making it a desirable choice for building reliable and
debuggable QA systems. We show that CBR-MRC provides high accuracy comparable
with large reader models and outperforms baselines by 11.5 and 8.4 EM on
NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability
of CBR-MRC in identifying not just the correct answer tokens but also the span
with the most relevant supporting evidence. Lastly, we observe that contexts
for certain question types show higher lexical diversity than others and find
that CBR-MRC is robust to these variations while performance using
fully-parametric methods drops.
