  Long short-term memory (LSTM) recurrent neural networks (RNNs) have been
shown to give state-of-the-art performance on many speech recognition tasks, as
they are able to provide the learned dynamically changing contextual window of
all sequence history. On the other hand, the convolutional neural networks
(CNNs) have brought significant improvements to deep feed-forward neural
networks (FFNNs), as they are able to better reduce spectral variation in the
input signal. In this paper, a network architecture called as convolutional
recurrent neural network (CRNN) is proposed by combining the CNN and LSTM RNN.
In the proposed CRNNs, each speech frame, without adjacent context frames, is
organized as a number of local feature patches along the frequency axis, and
then a LSTM network is performed on each feature patch along the time axis. We
train and compare FFNNs, LSTM RNNs and the proposed LSTM CRNNs at various
number of configurations. Experimental results show that the LSTM CRNNs can
exceed state-of-the-art speech recognition performance.
