  Recurrent neural networks are known for their notorious exploding and
vanishing gradient problem (EVGP). This problem becomes more evident in tasks
where the information needed to correctly solve them exist over long time
scales, because EVGP prevents important gradient components from being
back-propagated adequately over a large number of steps. We introduce a simple
stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization
and targeted towards addressing this problem. Specifically, we show that when
the LSTM weights are large, the gradient components through the linear path
(cell state) in the LSTM computational graph get suppressed. Based on the
hypothesis that these components carry information about long term dependencies
(which we show empirically), their suppression can prevent LSTMs from capturing
them. Our algorithm\footnote{Our code is available at
https://github.com/bhargav104/h-detach.} prevents gradients flowing through
this path from getting suppressed, thus allowing the LSTM to capture such
dependencies better. We show significant improvements over vanilla LSTM
gradient based training in terms of convergence speed, robustness to seed and
learning rate, and generalization using our modification of LSTM gradient on
various benchmark datasets.
