  Attention-based sequence-to-sequence (seq2seq) models have achieved promising
results in automatic speech recognition (ASR). However, as these models decode
in a left-to-right way, they do not have access to context on the right. We
leverage both left and right context by applying BERT as an external language
model to seq2seq ASR through knowledge distillation. In our proposed method,
BERT generates soft labels to guide the training of seq2seq ASR. Furthermore,
we leverage context beyond the current utterance as input to BERT. Experimental
evaluations show that our method significantly improves the ASR performance
from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ).
Knowledge distillation from BERT outperforms that from a transformer LM that
only looks at left context. We also show the effectiveness of leveraging
context beyond the current utterance. Our method outperforms other LM
application approaches such as n-best rescoring and shallow fusion, while it
does not require extra inference cost.
