  Although Reinforcement Learning (RL) algorithms have found tremendous success
in simulated domains, they often cannot directly be applied to physical
systems, especially in cases where there are hard constraints to satisfy (e.g.
on safety or resources). In standard RL, the agent is incentivized to explore
any behavior as long as it maximizes rewards, but in the real world, undesired
behavior can damage either the system or the agent in a way that breaks the
learning process itself. In this work, we model the problem of learning with
constraints as a Constrained Markov Decision Process and provide a new
on-policy formulation for solving it. A key contribution of our approach is to
translate cumulative cost constraints into state-based constraints. Through
this, we define a safe policy improvement method which maximizes returns while
ensuring that the constraints are satisfied at every step. We provide
theoretical guarantees under which the agent converges while ensuring safety
over the course of training. We also highlight the computational advantages of
this approach. The effectiveness of our approach is demonstrated on safe
navigation tasks and in safety-constrained versions of MuJoCo environments,
with deep neural networks.
