  Viewing recurrent neural networks (RNNs) as continuous-time dynamical
systems, we propose a recurrent unit that describes the hidden state's
evolution with two parts: a well-understood linear component plus a Lipschitz
nonlinearity. This particular functional form facilitates stability analysis of
the long-term behavior of the recurrent unit using tools from nonlinear systems
theory. In turn, this enables architectural design decisions before
experimentation. Sufficient conditions for global stability of the recurrent
unit are obtained, motivating a novel scheme for constructing hidden-to-hidden
matrices. Our experiments demonstrate that the Lipschitz RNN can outperform
existing recurrent units on a range of benchmark tasks, including computer
vision, language modeling and speech prediction tasks. Finally, through
Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more
robust with respect to input and parameter perturbations as compared to other
continuous-time RNNs.
