  Ensuring safety is important for the practical deployment of reinforcement
learning (RL). Various challenges must be addressed, such as handling
stochasticity in the environments, providing rigorous guarantees of persistent
state-wise safety satisfaction, and avoiding overly conservative behaviors that
sacrifice performance. We propose a new framework, Reachability Estimation for
Safe Policy Optimization (RESPO), for safety-constrained RL in general
stochastic settings. In the feasible set where there exist violation-free
policies, we optimize for rewards while maintaining persistent safety. Outside
this feasible set, our optimization produces the safest behavior by
guaranteeing entrance into the feasible set whenever possible with the least
cumulative discounted violations. We introduce a class of algorithms using our
novel reachability estimation function to optimize in our proposed framework
and in similar frameworks such as those concurrently handling multiple hard and
soft constraints. We theoretically establish that our algorithms almost surely
converge to locally optimal policies of our safe optimization framework. We
evaluate the proposed methods on a diverse suite of safe RL environments from
Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both
reward performance and safety compared with state-of-the-art baselines.
