  Deep reinforcement learning (RL) excels in various control tasks, yet the
absence of safety guarantees hampers its real-world applicability. In
particular, explorations during learning usually results in safety violations,
while the RL agent learns from those mistakes. On the other hand, safe control
techniques ensure persistent safety satisfaction but demand strong priors on
system dynamics, which is usually hard to obtain in practice. To address these
problems, we present Safe Set Guided State-wise Constrained Policy Optimization
(S-3PO), a pioneering algorithm generating state-wise safe optimal policies
with zero training violations, i.e., learning without mistakes. S-3PO first
employs a safety-oriented monitor with black-box dynamics to ensure safe
exploration. It then enforces a unique cost for the RL agent to converge to
optimal behaviors within safety constraints. S-3PO outperforms existing methods
in high-dimensional robotics tasks, managing state-wise constraints with zero
training violation. This innovation marks a significant stride towards
real-world safe RL deployment.
