  Scores from traditional confidence classifiers (CCs) in automatic speech
recognition (ASR) systems lack universal interpretation and vary with updates
to the underlying confidence or acoustic models (AMs). In this work, we build
interpretable confidence scores with an objective to closely align with ASR
accuracy. We propose a new sequence-level CC with a richer context providing CC
scores highly correlated with ASR accuracy and scores stable across CC updates.
Hence, expanding CC applications. Recently, AM customization has gained
traction with the widespread use of unified models. Conventional adaptation
strategies that customize AM expect well-matched data for the target domain
with gold-standard transcriptions. We propose a cost-effective method of using
CC scores to select an optimal adaptation data set, where we maximize ASR gains
from minimal data. We study data in various confidence ranges and optimally
choose data for AM adaptation with KL-Divergence regularization. On the
Microsoft voice search task, data selection for supervised adaptation using the
sequence-level confidence scores achieves word error rate reduction (WERR) of
8.5% for row-convolution LSTM (RC-LSTM) and 5.2% for latency-controlled
bidirectional LSTM (LC-BLSTM). In the semi-supervised case, with ASR hypotheses
as labels, our method provides WERR of 5.9% and 2.8% for RC-LSTM and LC-BLSTM,
respectively.
