  Reinforcement learning is showing great potentials in robotics applications,
including autonomous driving, robot manipulation and locomotion. However, with
complex uncertainties in the real-world environment, it is difficult to
guarantee the successful generalization and sim-to-real transfer of learned
policies theoretically. In this paper, we introduce and extend the idea of
robust stability and $H_\infty$ control to design policies with both stability
and robustness guarantee. Specifically, a sample-based approach for analyzing
the Lyapunov stability and performance robustness of a learning-based control
system is proposed. Based on the theoretical results, a maximum entropy
algorithm is developed for searching Lyapunov function and designing a policy
with provable robust stability guarantee. Without any specific domain
knowledge, our method can find a policy that is robust to various uncertainties
and generalizes well to different test environments. In our experiments, we
show that our method achieves better robustness to both large impulsive
disturbances and parametric variations in the environment than the state-of-art
results in both robust and generic RL, as well as classic control. Anonymous
code is available to reproduce the experimental results at
https://github.com/RobustStabilityGuaranteeRL/RobustStabilityGuaranteeRL.
