  In this paper, we propose a novel neural network structure, namely
\emph{feedforward sequential memory networks (FSMN)}, to model long-term
dependency in time series without using recurrent feedback. The proposed FSMN
is a standard fully-connected feedforward neural network equipped with some
learnable memory blocks in its hidden layers. The memory blocks use a
tapped-delay line structure to encode the long context information into a
fixed-size representation as short-term memory mechanism. We have evaluated the
proposed FSMNs in several standard benchmark tasks, including speech
recognition and language modelling. Experimental results have shown FSMNs
significantly outperform the conventional recurrent neural networks (RNN),
including LSTMs, in modeling sequential signals like speech or language.
Moreover, FSMNs can be learned much more reliably and faster than RNNs or LSTMs
due to the inherent non-recurrent model structure.
