  Recurrent neural networks (RNNs) provide state-of-the-art performance in
processing sequential data but are memory intensive to train, limiting the
flexibility of RNN models which can be trained. Reversible RNNs---RNNs for
which the hidden-to-hidden transition can be reversed---offer a path to reduce
the memory requirements of training, as hidden states need not be stored and
instead can be recomputed during backpropagation. We first show that perfectly
reversible RNNs, which require no storage of the hidden activations, are
fundamentally limited because they cannot forget information from their hidden
state. We then provide a scheme for storing a small number of bits in order to
allow perfect reversal with forgetting. Our method achieves comparable
performance to traditional models while reducing the activation memory cost by
a factor of 10--15. We extend our technique to attention-based
sequence-to-sequence models, where it maintains performance while reducing
activation memory cost by a factor of 5--10 in the encoder, and a factor of
10--15 in the decoder.
