  Time Delay Neural Networks (TDNNs) are widely used in both DNN-HMM based
hybrid speech recognition systems and recent end-to-end systems. Nevertheless,
the receptive fields of TDNNs are limited and fixed, which is not desirable for
tasks like speech recognition, where the temporal dynamics of speech are varied
and affected by many factors. This paper proposes to use deformable TDNNs for
adaptive temporal dynamics modeling in end-to-end speech recognition. Inspired
by deformable ConvNets, deformable TDNNs augment the temporal sampling
locations with additional offsets and learn the offsets automatically based on
the ASR criterion, without additional supervision. Experiments show that
deformable TDNNs obtain state-of-the-art results on WSJ benchmarks
(1.42\%/3.45\% WER on WSJ eval92/dev93 respectively), outperforming standard
TDNNs significantly. Furthermore, we propose the latency control mechanism for
deformable TDNNs, which enables deformable TDNNs to do streaming ASR without
accuracy degradation.
