  The performance of automatic speech recognition (ASR) has improved
tremendously due to the application of deep neural networks (DNNs). Despite
this progress, building a new ASR system remains a challenging task, requiring
various resources, multiple training stages and significant expertise. This
paper presents our Eesen framework which drastically simplifies the existing
pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen
involves learning a single recurrent neural network (RNN) predicting
context-independent targets (phonemes or characters). To remove the need for
pre-generated frame labels, we adopt the connectionist temporal classification
(CTC) objective function to infer the alignments between speech and label
sequences. A distinctive feature of Eesen is a generalized decoding approach
based on weighted finite-state transducers (WFSTs), which enables the efficient
incorporation of lexicons and language models into CTC decoding. Experiments
show that compared with the standard hybrid DNN systems, Eesen achieves
comparable word error rates (WERs), while at the same time speeding up decoding
significantly.
