  Evolution Strategies (ESs) have recently become popular for training deep
neural networks, in particular on reinforcement learning tasks, a special form
of controller design. Compared to classic problems in continuous direct search,
deep networks pose extremely high-dimensional optimization problems, with many
thousands or even millions of variables. In addition, many control problems
give rise to a stochastic fitness function. Considering the relevance of the
application, we study the suitability of evolution strategies for
high-dimensional, stochastic problems. Our results give insights into which
algorithmic mechanisms of modern ES are of value for the class of problems at
hand, and they reveal principled limitations of the approach. They are in line
with our theoretical understanding of ESs. We show that combining ESs that
offer reduced internal algorithm cost with uncertainty handling techniques
yields promising methods for this class of problems.
