  Second-pass rescoring is an important component in automatic speech
recognition (ASR) systems that is used to improve the outputs from a first-pass
decoder by implementing a lattice rescoring or $n$-best re-ranking. While
pretraining with a masked language model (MLM) objective has received great
success in various natural language understanding (NLU) tasks, it has not
gained traction as a rescoring model for ASR. Specifically, training a
bidirectional model like BERT on a discriminative objective such as minimum WER
(MWER) has not been explored. Here we show how to train a BERT-based rescoring
model with MWER loss, to incorporate the improvements of a discriminative loss
into fine-tuning of deep bidirectional pretrained models for ASR. Specifically,
we propose a fusion strategy that incorporates the MLM into the discriminative
training process to effectively distill knowledge from a pretrained model. We
further propose an alternative discriminative loss. This approach, which we
call RescoreBERT, reduces WER by 6.6%/3.4% relative on the LibriSpeech
clean/other test sets over a BERT baseline without discriminative objective. We
also evaluate our method on an internal dataset from a conversational agent and
find that it reduces both latency and WER (by 3 to 8% relative) over an LSTM
rescoring model.
