  Do question answering (QA) modeling improvements (e.g., choice of
architecture and training procedure) hold consistently across the diverse
landscape of QA benchmarks? To study this question, we introduce the notion of
concurrence -- two benchmarks have high concurrence on a set of modeling
approaches if they rank the modeling approaches similarly. We measure the
concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches
and find that human-constructed benchmarks have high concurrence amongst
themselves, even if their passage and question distributions are very
different. Surprisingly, even downsampled human-constructed benchmarks (i.e.,
collecting less data) and programmatically-generated benchmarks (e.g.,
cloze-formatted examples) have high concurrence with human-constructed
benchmarks. These results indicate that, despite years of intense community
focus on a small number of benchmarks, the modeling improvements studied hold
broadly.
