  Methods that extract policy primitives from offline demonstrations using deep
generative models have shown promise at accelerating reinforcement learning(RL)
for new tasks. Intuitively, these methods should also help to trainsafeRLagents
because they enforce useful skills. However, we identify these techniques are
not well equipped for safe policy learning because they ignore negative
experiences(e.g., unsafe or unsuccessful), focusing only on positive
experiences, which harms their ability to generalize to new tasks safely.
Rather, we model the latentsafetycontextusing principled contrastive training
on an offline dataset of demonstrations from many tasks, including both
negative and positive experiences. Using this late variable, our RL framework,
SAFEty skill pRiors (SAFER) extracts task-specific safe primitive skills to
safely and successfully generalize to new tasks. In the inference stage,
policies trained with SAFER learn to compose safe skills into successful
policies. We theoretically characterize why SAFER can enforce safe policy
learning and demonstrate its effectiveness on several complex safety-critical
robotic grasping tasks inspired by the game Operation, in which
SAFERoutperforms state-of-the-art primitive learning methods in success and
safety.
