  Several low-bandwidth distributable black-box optimization algorithms in the
family of finite differences such as Evolution Strategies have recently been
shown to perform nearly as well as tailored Reinforcement Learning methods in
some Reinforcement Learning domains. One shortcoming of these black-box methods
is that they must collect information about the structure of the return
function at every update, and can often employ only information drawn from a
distribution centered around the current parameters. As a result, when these
algorithms are distributed across many machines, a significant portion of total
runtime may be spent with many machines idle, waiting for a final return and
then for an update to be calculated. In this work we introduce a novel method
to use older data in finite difference algorithms, which produces a scalable
algorithm that avoids significant idle time or wasted computation.
