  Confidence estimation, in which we estimate the reliability of each
recognized token (e.g., word, sub-word, and character) in automatic speech
recognition (ASR) hypotheses and detect incorrectly recognized tokens, is an
important function for developing ASR applications. In this study, we perform
confidence estimation for end-to-end (E2E) ASR hypotheses. Recent E2E ASR
systems show high performance (e.g., around 5% token error rates) for various
ASR tasks. In such situations, confidence estimation becomes difficult since we
need to detect infrequent incorrect tokens from mostly correct token sequences.
To tackle this imbalanced dataset problem, we employ a bidirectional long
short-term memory (BLSTM)-based model as a strong binary-class
(correct/incorrect) sequence labeler that is trained with a class balancing
objective. We experimentally confirmed that, by utilizing several types of ASR
decoding scores as its auxiliary features, the model steadily shows high
confidence estimation performance under highly imbalanced settings. We also
confirmed that the BLSTM-based model outperforms Transformer-based confidence
estimation models, which greatly underestimate incorrect tokens.
