  Recently, end-to-end speech recognition with a hybrid model consisting of the
connectionist temporal classification(CTC) and the attention encoder-decoder
achieved state-of-the-art results. In this paper, we propose a novel CTC
decoder structure based on the experiments we conducted and explore the
relation between decoding performance and the depth of encoder. We also apply
attention smoothing mechanism to acquire more context information for
subword-based decoding. Taken together, these strategies allow us to achieve a
word error rate(WER) of 4.43% without LM and 3.34% with RNN-LM on the
test-clean subset of the LibriSpeech corpora, which by far are the best
reported WERs for end-to-end ASR systems on this dataset.
