  This study presents a novel approach for knowledge distillation (KD) from a
BERT teacher model to an automatic speech recognition (ASR) model using
intermediate layers. To distil the teacher's knowledge, we use an attention
decoder that learns from BERT's token probabilities. Our method shows that
language model (LM) information can be more effectively distilled into an ASR
model using both the intermediate layers and the final layer. By using the
intermediate layers as distillation target, we can more effectively distil LM
knowledge into the lower network layers. Using our method, we achieve better
recognition accuracy than with shallow fusion of an external LM, allowing us to
maintain fast parallel decoding. Experiments on the LibriSpeech dataset
demonstrate the effectiveness of our approach in enhancing greedy decoding with
connectionist temporal classification (CTC).
