  Recurrent neural networks (RNNs) are particularly well-suited for modeling
long-term dependencies in sequential data, but are notoriously hard to train
because the error backpropagated in time either vanishes or explodes at an
exponential rate. While a number of works attempt to mitigate this effect
through gated recurrent units, well-chosen parametric constraints, and
skip-connections, we develop a novel perspective that seeks to evolve the
hidden state on the equilibrium manifold of an ordinary differential equation
(ODE). We propose a family of novel RNNs, namely {\em Equilibriated Recurrent
Neural Networks} (ERNNs) that overcome the gradient decay or explosion effect
and lead to recurrent models that evolve on the equilibrium manifold. We show
that equilibrium points are stable, leading to fast convergence of the
discretized ODE to fixed points. Furthermore, ERNNs account for long-term
dependencies, and can efficiently recall informative aspects of data from the
distant past. We show that ERNNs achieve state-of-the-art accuracy on many
challenging data sets with 3-10x speedups, 1.5-3x model size reduction, and
with similar prediction cost relative to vanilla RNNs.
