  In this paper, we present an end-to-end training framework for building
state-of-the-art end-to-end speech recognition systems. Our training system
utilizes a cluster of Central Processing Units(CPUs) and Graphics Processing
Units (GPUs). The entire data reading, large scale data augmentation, neural
network parameter updates are all performed "on-the-fly". We use vocal tract
length perturbation [1] and an acoustic simulator [2] for data augmentation.
The processed features and labels are sent to the GPU cluster. The Horovod
allreduce approach is employed to train neural network parameters. We evaluated
the effectiveness of our system on the standard Librispeech corpus [3] and the
10,000-hr anonymized Bixby English dataset. Our end-to-end speech recognition
system built using this training infrastructure showed a 2.44 % WER on
test-clean of the LibriSpeech test set after applying shallow fusion with a
Transformer language model (LM). For the proprietary English Bixby open domain
test set, we obtained a WER of 7.92 % using a Bidirectional Full Attention
(BFA) end-to-end model after applying shallow fusion with an RNN-LM. When the
monotonic chunckwise attention (MoCha) based approach is employed for streaming
speech recognition, we obtained a WER of 9.95 % on the same Bixby open domain
test set.
