  We introduce a constrained optimization method for policy gradient
reinforcement learning, which uses a virtual trust region to regulate each
policy update. In addition to using the proximity of one single old policy as
the normal trust region, we propose forming a second trust region through
another virtual policy representing a wide range of past policies. We then
enforce the new policy to stay closer to the virtual policy, which is
beneficial if the old policy performs poorly. More importantly, we propose a
mechanism to automatically build the virtual policy from a memory of past
policies, providing a new capability for dynamically learning appropriate
virtual trust regions during the optimization process. Our proposed method,
dubbed Memory-Constrained Policy Optimization (MCPO), is examined in diverse
environments, including robotic locomotion control, navigation with sparse
rewards and Atari games, consistently demonstrating competitive performance
against recent on-policy constrained policy gradient methods.
