  We study continuous action reinforcement learning problems in which it is
crucial that the agent interacts with the environment only through safe
policies, i.e.,~policies that do not take the agent to undesirable situations.
We formulate these problems as constrained Markov decision processes (CMDPs)
and present safe policy optimization algorithms that are based on a Lyapunov
approach to solve them. Our algorithms can use any standard policy gradient
(PG) method, such as deep deterministic policy gradient (DDPG) or proximal
policy optimization (PPO), to train a neural network policy, while guaranteeing
near-constraint satisfaction for every policy update by projecting either the
policy parameter or the action onto the set of feasible solutions induced by
the state-dependent linearized Lyapunov constraints. Compared to the existing
constrained PG algorithms, ours are more data efficient as they are able to
utilize both on-policy and off-policy data. Moreover, our action-projection
algorithm often leads to less conservative policy updates and allows for
natural integration into an end-to-end PG training pipeline. We evaluate our
algorithms and compare them with the state-of-the-art baselines on several
simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation
problem, demonstrating their effectiveness in terms of balancing performance
and constraint satisfaction. Videos of the experiments can be found in the
following link:
https://drive.google.com/file/d/1pzuzFqWIE710bE2U6DmS59AfRzqK2Kek/view?usp=sharing.
