  The effectiveness of recurrent neural networks can be largely influenced by
their ability to store into their dynamical memory information extracted from
input sequences at different frequencies and timescales. Such a feature can be
introduced into a neural architecture by an appropriate modularization of the
dynamic memory. In this paper we propose a novel incrementally trained
recurrent architecture targeting explicitly multi-scale learning. First, we
show how to extend the architecture of a simple RNN by separating its hidden
state into different modules, each subsampling the network hidden activations
at different frequencies. Then, we discuss a training algorithm where new
modules are iteratively added to the model to learn progressively longer
dependencies. Each new module works at a slower frequency than the previous
ones and it is initialized to encode the subsampled sequence of hidden
activations. Experimental results on synthetic and real-world datasets on
speech recognition and handwritten characters show that the modular
architecture and the incremental training algorithm improve the ability of
recurrent neural networks to capture long-term dependencies.
