  Racing demands each vehicle to drive at its physical limits, when any safety
infraction could lead to catastrophic failure. In this work, we study the
problem of safe reinforcement learning (RL) for autonomous racing, using the
vehicle's ego-camera view and speed as input. Given the nature of the task,
autonomous agents need to be able to 1) identify and avoid unsafe scenarios
under the complex vehicle dynamics, and 2) make sub-second decision in a
fast-changing environment. To satisfy these criteria, we propose to incorporate
Hamilton-Jacobi (HJ) reachability theory, a safety verification method for
general non-linear systems, into the constrained Markov decision process (CMDP)
framework. HJ reachability not only provides a control-theoretic approach to
learn about safety, but also enables low-latency safety verification. Though HJ
reachability is traditionally not scalable to high-dimensional systems, we
demonstrate that with neural approximation, the HJ safety value can be learned
directly on vision context -- the highest-dimensional problem studied via the
method, to-date. We evaluate our method on several benchmark tasks, including
Safety Gym and Learn-to-Race (L2R), a recently-released high-fidelity
autonomous racing environment. Our approach has significantly fewer constraint
violations in comparison to other constrained RL baselines in Safety Gym, and
achieves the new state-of-the-art results on the L2R benchmark task. We provide
additional visualization of agent behavior at the following anonymized paper
website: https://sites.google.com/view/safeautonomousracing/home
