  We propose a simple yet effective method to compress an RNN-Transducer
(RNN-T) through the well-known knowledge distillation paradigm. We show that
the transducer's encoder outputs naturally have a high entropy and contain rich
information about acoustically similar word-piece confusions. This rich
information is suppressed when combined with the lower entropy decoder outputs
to produce the joint network logits. Consequently, we introduce an auxiliary
loss to distill the encoder logits from a teacher transducer's encoder, and
explore training strategies where this encoder distillation works effectively.
We find that tandem training of teacher and student encoders with an inplace
encoder distillation outperforms the use of a pre-trained and static teacher
transducer. We also report an interesting phenomenon we refer to as implicit
distillation, that occurs when the teacher and student encoders share the same
decoder. Our experiments show 5.37-8.4% relative word error rate reductions
(WERR) on in-house test sets, and 5.05-6.18% relative WERRs on LibriSpeech test
sets.
