  Reinforcement learning (RL) has emerged as a promising paradigm in complex
and continuous robotic tasks, however, safe exploration has been one of the
main challenges, especially in contact-rich manipulation tasks in unstructured
environments. Focusing on this issue, we propose SRL-VIC: a model-free safe RL
framework combined with a variable impedance controller (VIC). Specifically,
safety critic and recovery policy networks are pre-trained where safety critic
evaluates the safety of the next action using a risk value before it is
executed and the recovery policy suggests a corrective action if the risk value
is high. Furthermore, the policies are updated online where the task policy not
only achieves the task but also modulates the stiffness parameters to keep a
safe and compliant profile. A set of experiments in contact-rich maze tasks
demonstrate that our framework outperforms the baselines (without the recovery
mechanism and without the VIC), yielding a good trade-off between efficient
task accomplishment and safety guarantee. We show our policy trained on
simulation can be deployed on a physical robot without fine-tuning, achieving
successful task completion with robustness and generalization. The video is
available at https://youtu.be/ksWXR3vByoQ.
