  Sequence-to-sequence models have shown success in end-to-end speech
recognition. However these models have only used shallow acoustic encoder
networks. In our work, we successively train very deep convolutional networks
to add more expressive power and better generalization for end-to-end ASR
models. We apply network-in-network principles, batch normalization, residual
connections and convolutional LSTMs to build very deep recurrent and
convolutional structures. Our models exploit the spectral structure in the
feature space and add computational depth without overfitting issues. We
experiment with the WSJ ASR task and achieve 10.5\% word error rate without any
dictionary or language using a 15 layer deep network.
