  We study constrained reinforcement learning (CRL) from a novel perspective by
setting constraints directly on state density functions, rather than the value
functions considered by previous works. State density has a clear physical and
mathematical interpretation, and is able to express a wide variety of
constraints such as resource limits and safety requirements. Density
constraints can also avoid the time-consuming process of designing and tuning
cost functions required by value function-based constraints to encode system
specifications. We leverage the duality between density functions and Q
functions to develop an effective algorithm to solve the density constrained RL
problem optimally and the constrains are guaranteed to be satisfied. We prove
that the proposed algorithm converges to a near-optimal solution with a bounded
error even when the policy update is imperfect. We use a set of comprehensive
experiments to demonstrate the advantages of our approach over state-of-the-art
CRL methods, with a wide range of density constrained tasks as well as standard
CRL benchmarks such as Safety-Gym.
