  Solving tasks in Reinforcement Learning is no easy feat. As the goal of the
agent is to maximize the accumulated reward, it often learns to exploit
loopholes and misspecifications in the reward signal resulting in unwanted
behavior. While constraints may solve this issue, there is no closed form
solution for general constraints. In this work we present a novel
multi-timescale approach for constrained policy optimization, called `Reward
Constrained Policy Optimization' (RCPO), which uses an alternative penalty
signal to guide the policy towards a constraint satisfying one. We prove the
convergence of our approach and provide empirical evidence of its ability to
train constraint satisfying policies.
