  Connectionist Temporal Classification (CTC) suffers from the latency problem
when applied to streaming models. We argue that in CTC lattice, the alignments
that can access more future context are preferred during training, thereby
leading to higher symbol delay. In this work we propose the delay-penalized CTC
which is augmented with latency penalty regularization. We devise a flexible
and efficient implementation based on the differentiable Finite State
Transducer (FST). Specifically, by attaching a binary attribute to CTC
topology, we can locate the frames that firstly emit non-blank tokens on the
resulting CTC lattice, and add the frame offsets to the log-probabilities.
Experimental results demonstrate the effectiveness of our proposed
delay-penalized CTC, which is able to balance the delay-accuracy trade-off.
Furthermore, combining the delay-penalized transducer enables the CTC model to
achieve better performance and lower latency. Our work is open-sourced and
publicly available https://github.com/k2-fsa/k2.
