  Reinforcement Learning (RL) for constrained MDPs (CMDPs) is an increasingly
important problem for various applications. Often, the average criterion is
more suitable than the discounted criterion. Yet, RL for average-CMDPs (ACMDPs)
remains a challenging problem. Algorithms designed for discounted constrained
RL problems often do not perform well for the average CMDP setting. In this
paper, we introduce a new policy optimization with function approximation
algorithm for constrained MDPs with the average criterion. The
Average-Constrained Policy Optimization (ACPO) algorithm is inspired by trust
region-based policy optimization algorithms. We develop basic sensitivity
theory for average CMDPs, and then use the corresponding bounds in the design
of the algorithm. We provide theoretical guarantees on its performance, and
through extensive experimental work in various challenging OpenAI Gym
environments, show its superior empirical performance when compared to other
state-of-the-art algorithms adapted for the ACMDPs.
