  Learning a risk-aware policy is essential but rather challenging in
unstructured robotic tasks. Safe reinforcement learning methods open up new
possibilities to tackle this problem. However, the conservative policy updates
make it intractable to achieve sufficient exploration and desirable performance
in complex, sample-expensive environments. In this paper, we propose a
dual-agent safe reinforcement learning strategy consisting of a baseline and a
safe agent. Such a decoupled framework enables high flexibility, data
efficiency and risk-awareness for RL-based control. Concretely, the baseline
agent is responsible for maximizing rewards under standard RL settings. Thus,
it is compatible with off-the-shelf training techniques of unconstrained
optimization, exploration and exploitation. On the other hand, the safe agent
mimics the baseline agent for policy improvement and learns to fulfill safety
constraints via off-policy RL tuning. In contrast to training from scratch,
safe policy correction requires significantly fewer interactions to obtain a
near-optimal policy. The dual policies can be optimized synchronously via a
shared replay buffer, or leveraging the pre-trained model or the
non-learning-based controller as a fixed baseline agent. Experimental results
show that our approach can learn feasible skills without prior knowledge as
well as deriving risk-averse counterparts from pre-trained unsafe policies. The
proposed method outperforms the state-of-the-art safe RL algorithms on
difficult robot locomotion and manipulation tasks with respect to both safety
constraint satisfaction and sample efficiency.
