  Sequential information contains short- to long-range dependencies; however,
learning long-timescale information has been a challenge for recurrent neural
networks. Despite improvements in long short-term memory networks (LSTMs), the
forgetting mechanism results in the exponential decay of information, limiting
their capacity to capture long-timescale information. Here, we propose a power
law forget gate, which instead learns to forget information along a slower
power law decay function. Specifically, the new gate learns to control the
power law decay factor, p, allowing the network to adjust the information decay
rate according to task demands. Our experiments show that an LSTM with power
law forget gates (pLSTM) can effectively capture long-range dependencies beyond
hundreds of elements on image classification, language modeling, and
categorization tasks, improving performance over the vanilla LSTM. We also
inspected the revised forget gate by varying the initialization of p, setting p
to a fixed value, and ablating cells in the pLSTM network. The results show
that the information decay can be controlled by the learnable decay factor p,
which allows pLSTM to achieve its superior performance. Altogether, we found
that LSTM with the proposed forget gate can learn long-term dependencies,
outperforming other recurrent networks in multiple domains; such gating
mechanism can be integrated into other architectures for improving the learning
of long timescale information in recurrent neural networks.
