  Constrained Reinforcement Learning has been employed to enforce safety
constraints on policy through the use of expected cost constraints. The key
challenge is in handling expected cost accumulated using the policy and not
just in a single step. Existing methods have developed innovative ways of
converting this cost constraint over entire policy to constraints over local
decisions (at each time step). While such approaches have provided good
solutions with regards to objective, they can either be overly aggressive or
conservative with respect to costs. This is owing to use of estimates for
"future" or "backward" costs in local cost constraints.
  To that end, we provide an equivalent unconstrained formulation to
constrained RL that has an augmented state space and reward penalties. This
intuitive formulation is general and has interesting theoretical properties.
More importantly, this provides a new paradigm for solving constrained RL
problems effectively. As we show in our experimental results, we are able to
outperform leading approaches on multiple benchmark problems from literature.
