  Some of the most successful applications of deep reinforcement learning to
challenging domains in discrete and continuous control have used policy
gradient methods in the on-policy setting. However, policy gradients can suffer
from large variance that may limit performance, and in practice require
carefully tuned entropy regularization to prevent policy collapse. As an
alternative to policy gradient algorithms, we introduce V-MPO, an on-policy
adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs
policy iteration based on a learned state-value function. We show that V-MPO
surpasses previously reported scores for both the Atari-57 and DMLab-30
benchmark suites in the multi-task setting, and does so reliably without
importance weighting, entropy regularization, or population-based tuning of
hyperparameters. On individual DMLab and Atari levels, the proposed algorithm
can achieve scores that are substantially higher than has previously been
reported. V-MPO is also applicable to problems with high-dimensional,
continuous action spaces, which we demonstrate in the context of learning to
control simulated humanoids with 22 degrees of freedom from full state
observations and 56 degrees of freedom from pixel observations, as well as
example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic
scores than previously reported.
