  Sequence transducers, such as the RNN-T and the Conformer-T, are one of the
most promising models of end-to-end speech recognition, especially in streaming
scenarios where both latency and accuracy are important. Although various
methods, such as alignment-restricted training and FastEmit, have been studied
to reduce the latency, latency reduction is often accompanied with a
significant degradation in accuracy. We argue that this suboptimal performance
might be caused because none of the prior methods explicitly model and reduce
the latency. In this paper, we propose a new training method to explicitly
model and reduce the latency of sequence transducer models. First, we define
the expected latency at each diagonal line on the lattice, and show that its
gradient can be computed efficiently within the forward-backward algorithm.
Then we augment the transducer loss with this expected latency, so that an
optimal trade-off between latency and accuracy is achieved. Experimental
results on the WSJ dataset show that the proposed minimum latency training
reduces the latency of causal Conformer-T from 220 ms to 27 ms within a WER
degradation of 0.7%, and outperforms conventional alignment-restricted training
(110 ms) and FastEmit (67 ms) methods.
