  Posing reading comprehension as a generation problem provides a great deal of
flexibility, allowing for open-ended questions with few restrictions on
possible answers. However, progress is impeded by existing generation metrics,
which rely on token overlap and are agnostic to the nuances of reading
comprehension. To address this, we introduce a benchmark for training and
evaluating generative reading comprehension metrics: MOdeling Correctness with
Human Annotations. MOCHA contains 40K human judgement scores on model outputs
from 6 diverse question answering datasets and an additional set of minimal
pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for
Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms
baseline metrics by 10 to 36 absolute Pearson points on held-out annotations.
When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy,
outperforming baselines by 14 to 26 absolute percentage points while leaving
significant room for improvement. MOCHA presents a challenging problem for
developing accurate and robust generative reading comprehension metrics.
