  We address the problem of calibrating prediction confidence for output
entities of interest in natural language processing (NLP) applications. It is
important that NLP applications such as named entity recognition and question
answering produce calibrated confidence scores for their predictions,
especially if the system is to be deployed in a safety-critical domain such as
healthcare. However, the output space of such structured prediction models is
often too large to adapt binary or multi-class calibration methods directly. In
this study, we propose a general calibration scheme for output entities of
interest in neural-network based structured prediction models. Our proposed
method can be used with any binary class calibration scheme and a neural
network model. Additionally, we show that our calibration method can also be
used as an uncertainty-aware, entity-specific decoding step to improve the
performance of the underlying model at no additional training cost or data
requirements. We show that our method outperforms current calibration
techniques for named-entity-recognition, part-of-speech and question answering.
We also improve our model's performance from our decoding step across several
tasks and benchmark datasets. Our method improves the calibration and model
performance on out-of-domain test scenarios as well.
