  We say an algorithm is batch size-invariant if changes to the batch size can
largely be compensated for by changes to other hyperparameters. Stochastic
gradient descent is well-known to have this property at small batch sizes, via
the learning rate. However, some policy optimization algorithms (such as PPO)
do not have this property, because of how they control the size of policy
updates. In this work we show how to make these algorithms batch
size-invariant. Our key insight is to decouple the proximal policy (used for
controlling policy updates) from the behavior policy (used for off-policy
corrections). Our experiments help explain why these algorithms work, and
additionally show how they can make more efficient use of stale data.
