  Despite recent advances in training recurrent neural networks (RNNs),
capturing long-term dependencies in sequences remains a fundamental challenge.
Most approaches use backpropagation through time (BPTT), which is difficult to
scale to very long sequences. This paper proposes a simple method that improves
the ability to capture long term dependencies in RNNs by adding an unsupervised
auxiliary loss to the original objective. This auxiliary loss forces RNNs to
either reconstruct previous events or predict next events in a sequence, making
truncated backpropagation feasible for long sequences and also improving full
BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel
image classification with sequence lengths up to 16\,000, and a real document
classification benchmark. Our results highlight good performance and resource
efficiency of this approach over competitive baselines, including other
recurrent models and a comparable sized Transformer. Further analyses reveal
beneficial effects of the auxiliary loss on optimization and regularization, as
well as extreme cases where there is little to no backpropagation.
