  Reinforcement learning (RL) has achieved promising results on most robotic
control tasks. Safety of learning-based controllers is an essential notion of
ensuring the effectiveness of the controllers. Current methods adopt whole
consistency constraints during the training, thus resulting in inefficient
exploration in the early stage. In this paper, we propose an algorithm named
Constrained Policy Optimization with Extra Safety Budget (ESB-CPO) to strike a
balance between the exploration efficiency and the constraints satisfaction. In
the early stage, our method loosens the practical constraints of unsafe
transitions (adding extra safety budget) with the aid of a new metric we
propose. With the training process, the constraints in our optimization problem
become tighter. Meanwhile, theoretical analysis and practical experiments
demonstrate that our method gradually meets the cost limit's demand in the
final training stage. When evaluated on Safety-Gym and Bullet-Safety-Gym
benchmarks, our method has shown its advantages over baseline algorithms in
terms of safety and optimality. Remarkably, our method gains remarkable
performance improvement under the same cost limit compared with baselines.
