  Long short-term memory recurrent neural networks (LSTM-RNNs) are considered
state-of-the art in many speech processing tasks. The recurrence in the
network, in principle, allows any input to be remembered for an indefinite
time, a feature very useful for sequential data like speech. However, very
little is known about which information is actually stored in the LSTM and for
how long. We address this problem by using a memory reset approach which allows
us to evaluate network performance depending on the allowed memory time span.
We apply this approach to the task of multi-speaker source separation, but it
can be used for any task using RNNs. We find a strong performance effect of
short-term (shorter than 100 milliseconds) linguistic processes. Only speaker
characteristics are kept in the memory for longer than 400 milliseconds.
Furthermore, we confirm that performance-wise it is sufficient to implement
longer memory in deeper layers. Finally, in a bidirectional model, the backward
models contributes slightly more to the separation performance than the forward
model.
