  Deep learning has had a far reaching impact in robotics. Specifically, deep
reinforcement learning algorithms have been highly effective in synthesizing
neural-network controllers for a wide range of tasks. However, despite this
empirical success, these controllers still lack theoretical guarantees on their
performance, such as Lyapunov stability (i.e., all trajectories of the
closed-loop system are guaranteed to converge to a goal state under the control
policy). This is in stark contrast to traditional model-based controller
design, where principled approaches (like LQR) can synthesize stable
controllers with provable guarantees. To address this gap, we propose a generic
method to synthesize a Lyapunov-stable neural-network controller, together with
a neural-network Lyapunov function to simultaneously certify its stability. Our
approach formulates the Lyapunov condition verification as a mixed-integer
linear program (MIP). Our MIP verifier either certifies the Lyapunov condition,
or generates counter examples that can help improve the candidate controller
and the Lyapunov function. We also present an optimization program to compute
an inner approximation of the region of attraction for the closed-loop system.
We apply our approach to robots including an inverted pendulum, a 2D and a 3D
quadrotor, and showcase that our neural-network controller outperforms a
baseline LQR controller. The code is open sourced at
\url{https://github.com/StanfordASL/neural-network-lyapunov}.
