  We study calibration in question answering, estimating whether model
correctly predicts answer for each question. Unlike prior work which mainly
rely on the model's confidence score, our calibrator incorporates information
about the input example (e.g., question and the evidence context). Together
with data augmentation via back translation, our simple approach achieves 5-10%
gains in calibration accuracy on reading comprehension benchmarks. Furthermore,
we present the first calibration study in the open retrieval setting, comparing
the calibration accuracy of retrieval-based span prediction models and answer
generation models. Here again, our approach shows consistent gains over
calibrators relying on the model confidence. Our simple and efficient
calibrator can be easily adapted to many tasks and model architectures, showing
robust gains in all settings.
