  Most of reinforcement learning algorithms optimize the discounted criterion
which is beneficial to accelerate the convergence and reduce the variance of
estimates. Although the discounted criterion is appropriate for certain tasks
such as financial related problems, many engineering problems treat future
rewards equally and prefer a long-run average criterion. In this paper, we
study the reinforcement learning problem with the long-run average criterion.
Firstly, we develop a unified trust region theory with discounted and average
criteria and derive a novel performance bound within the trust region with the
Perturbation Analysis (PA) theory. Secondly, we propose a practical algorithm
named Average Policy Optimization (APO), which improves the value estimation
with a novel technique named Average Value Constraint. Finally, experiments are
conducted in the continuous control environment MuJoCo. In most tasks, APO
performs better than the discounted PPO, which demonstrates the effectiveness
of our approach. Our work provides a unified framework of the trust region
approach including both the discounted and average criteria, which may
complement the framework of reinforcement learning beyond the discounted
objectives.
