  Deep reinforcement learning (DRL) has demonstrated remarkable performance in
many continuous control tasks. However, a significant obstacle to the
real-world application of DRL is the lack of safety guarantees. Although DRL
agents can satisfy system safety in expectation through reward shaping,
designing agents to consistently meet hard constraints (e.g., safety
specifications) at every time step remains a formidable challenge. In contrast,
existing work in the field of safe control provides guarantees on persistent
satisfaction of hard safety constraints. However, these methods require
explicit analytical system dynamics models to synthesize safe control, which
are typically inaccessible in DRL settings. In this paper, we present a
model-free safe control algorithm, the implicit safe set algorithm, for
synthesizing safeguards for DRL agents that ensure provable safety throughout
training. The proposed algorithm synthesizes a safety index (barrier
certificate) and a subsequent safe control law solely by querying a black-box
dynamic function (e.g., a digital twin simulator). Moreover, we theoretically
prove that the implicit safe set algorithm guarantees finite time convergence
to the safe set and forward invariance for both continuous-time and
discrete-time systems. We validate the proposed algorithm on the
state-of-the-art Safety Gym benchmark, where it achieves zero safety violations
while gaining $95\% \pm 9\%$ cumulative reward compared to state-of-the-art
safe DRL methods. Furthermore, the resulting algorithm scales well to
high-dimensional systems with parallel computing.
