  While models have reached superhuman performance on popular question
answering (QA) datasets such as SQuAD, they have yet to outperform humans on
the task of question answering itself. In this paper, we investigate if models
are learning reading comprehension from QA datasets by evaluating BERT-based
models across five datasets. We evaluate models on their generalizability to
out-of-domain examples, responses to missing or incorrect data, and ability to
handle question variations. We find that no single dataset is robust to all of
our experiments and identify shortcomings in both datasets and evaluation
methods. Following our analysis, we make recommendations for building future QA
datasets that better evaluate the task of question answering through reading
comprehension. We also release code to convert QA datasets to a shared format
for easier experimentation at
https://github.com/amazon-research/qa-dataset-converter.
