  Constrained Markov Decision Process (CMDP) is a natural framework for
reinforcement learning tasks with safety constraints, where agents learn a
policy that maximizes the long-term reward while satisfying the constraints on
the long-term cost. A canonical approach for solving CMDPs is the primal-dual
method which updates parameters in primal and dual spaces in turn. Existing
methods for CMDPs only use on-policy data for dual updates, which results in
sample inefficiency and slow convergence. In this paper, we propose a policy
search method for CMDPs called Accelerated Primal-Dual Optimization (APDO),
which incorporates an off-policy trained dual variable in the dual update
procedure while updating the policy in primal space with on-policy likelihood
ratio gradient. Experimental results on a simulated robot locomotion task show
that APDO achieves better sample efficiency and faster convergence than
state-of-the-art approaches for CMDPs.
