  There has been considerable progress on academic benchmarks for the Reading
Comprehension (RC) task with State-of-the-Art models closing the gap with human
performance on extractive question answering. Datasets such as SQuAD 2.0 & NQ
have also introduced an auxiliary task requiring models to predict when a
question has no answer in the text. However, in production settings, it is also
necessary to provide confidence estimates for the performance of the underlying
RC model at both answer extraction and "answerability" detection. We propose a
novel post-prediction confidence estimation model, which we call Mr.C (short
for Mr. Confident), that can be trained to improve a system's ability to
refrain from making incorrect predictions with improvements of up to 4 points
as measured by Area Under the Curve (AUC) scores. Mr.C can benefit from a novel
white-box feature that leverages the underlying RC model's gradients.
Performance prediction is particularly important in cases of domain shift (as
measured by training RC models on SQUAD 2.0 and evaluating on NQ), where Mr.C
not only improves AUC, but also traditional answerability prediction (as
measured by a 5 point improvement in F1).
