  Learning to solve sequential tasks with recurrent models requires the ability
to memorize long sequences and to extract task-relevant features from them. In
this paper, we study the memorization subtask from the point of view of the
design and training of recurrent neural networks. We propose a new model, the
Linear Memory Network, which features an encoding-based memorization component
built with a linear autoencoder for sequences. We extend the memorization
component with a modular memory that encodes the hidden state sequence at
different sampling frequencies. Additionally, we provide a specialized training
algorithm that initializes the memory to efficiently encode the hidden
activations of the network. The experimental results on synthetic and
real-world datasets show that specializing the training algorithm to train the
memorization component always improves the final performance whenever the
memorization of long sequences is necessary to solve the problem.
