  Machine reading comprehension (MRC) has received considerable attention as a
benchmark for natural language understanding. However, the conventional task
design of MRC lacks explainability beyond the model interpretation, i.e.,
reading comprehension by a model cannot be explained in human terms. To this
end, this position paper provides a theoretical basis for the design of MRC
datasets based on psychology as well as psychometrics, and summarizes it in
terms of the prerequisites for benchmarking MRC. We conclude that future
datasets should (i) evaluate the capability of the model for constructing a
coherent and grounded representation to understand context-dependent situations
and (ii) ensure substantive validity by shortcut-proof questions and
explanation as a part of the task design.
