  Safety is essential for reinforcement learning (RL) applied in the real
world. Adding chance constraints (or probabilistic constraints) is a suitable
way to enhance RL safety under uncertainty. Existing chance-constrained RL
methods like the penalty methods and the Lagrangian methods either exhibit
periodic oscillations or learn an over-conservative or unsafe policy. In this
paper, we address these shortcomings by proposing a separated
proportional-integral Lagrangian (SPIL) algorithm. We first review the
constrained policy optimization process from a feedback control perspective,
which regards the penalty weight as the control input and the safe probability
as the control output. Based on this, the penalty method is formulated as a
proportional controller, and the Lagrangian method is formulated as an integral
controller. We then unify them and present a proportional-integral Lagrangian
method to get both their merits, with an integral separation technique to limit
the integral value in a reasonable range. To accelerate training, the gradient
of safe probability is computed in a model-based manner. We demonstrate our
method can reduce the oscillations and conservatism of RL policy in a
car-following simulation. To prove its practicality, we also apply our method
to a real-world mobile robot navigation task, where our robot successfully
avoids a moving obstacle with highly uncertain or even aggressive behaviors.
