  In reinforcement learning for safety-critical settings, it is often desirable
for the agent to obey safety constraints at all points in time, including
during training. We present a novel neurosymbolic approach called SPICE to
solve this safe exploration problem. SPICE uses an online shielding layer based
on symbolic weakest preconditions to achieve a more precise safety analysis
than existing tools without unduly impacting the training process. We evaluate
the approach on a suite of continuous control benchmarks and show that it can
achieve comparable performance to existing safe learning techniques while
incurring fewer safety violations. Additionally, we present theoretical results
showing that SPICE converges to the optimal safe policy under reasonable
assumptions.
