  In this paper, we present an improved feedforward sequential memory networks
(FSMN) architecture, namely Deep-FSMN (DFSMN), by introducing skip connections
between memory blocks in adjacent layers. These skip connections enable the
information flow across different layers and thus alleviate the gradient
vanishing problem when building very deep structure. As a result, DFSMN
significantly benefits from these skip connections and deep structure. We have
compared the performance of DFSMN to BLSTM both with and without lower frame
rate (LFR) on several large speech recognition tasks, including English and
Mandarin. Experimental results shown that DFSMN can consistently outperform
BLSTM with dramatic gain, especially trained with LFR using CD-Phone as
modeling units. In the 2000 hours Fisher (FSH) task, the proposed DFSMN can
achieve a word error rate of 9.4% by purely using the cross-entropy criterion
and decoding with a 3-gram language model, which achieves a 1.5% absolute
improvement compared to the BLSTM. In a 20000 hours Mandarin recognition task,
the LFR trained DFSMN can achieve more than 20% relative improvement compared
to the LFR trained BLSTM. Moreover, we can easily design the lookahead filter
order of the memory blocks in DFSMN to control the latency for real-time
applications.
