  Optimizing functions without access to gradients is the remit of black-box
methods such as evolution strategies. While highly general, their learning
dynamics are often times heuristic and inflexible - exactly the limitations
that meta-learning can address. Hence, we propose to discover effective update
rules for evolution strategies via meta-learning. Concretely, our approach
employs a search strategy parametrized by a self-attention-based architecture,
which guarantees the update rule is invariant to the ordering of the candidate
solutions. We show that meta-evolving this system on a small set of
representative low-dimensional analytic optimization problems is sufficient to
discover new evolution strategies capable of generalizing to unseen
optimization problems, population sizes and optimization horizons. Furthermore,
the same learned evolution strategy can outperform established neuroevolution
baselines on supervised and continuous control tasks. As additional
contributions, we ablate the individual neural network components of our
method; reverse engineer the learned strategy into an explicit heuristic form,
which remains highly competitive; and show that it is possible to
self-referentially train an evolution strategy from scratch, with the learned
update rule used to drive the outer meta-learning loop.
