  Although deep reinforcement learning methods can learn effective policies for
challenging problems such as Atari games and robotics tasks, algorithms are
complex, and training times are often long. This study investigates how
Evolution Strategies perform compared to gradient-based deep reinforcement
learning methods. We use Evolution Strategies to optimize the weights of a
neural network via neuroevolution, performing direct policy search. We
benchmark both deep policy networks and networks consisting of a single linear
layer from observations to actions for three gradient-based methods, such as
Proximal Policy Optimization. These methods are evaluated against three
classical Evolution Strategies and Augmented Random Search, which all use
linear policy networks. Our results reveal that Evolution Strategies can find
effective linear policies for many reinforcement learning benchmark tasks,
unlike deep reinforcement learning methods that can only find successful
policies using much larger networks, suggesting that current benchmarks are
easier to solve than previously assumed. Interestingly, Evolution Strategies
also achieve results comparable to gradient-based deep reinforcement learning
algorithms for higher-complexity tasks. Furthermore, we find that by directly
accessing the memory state of the game, Evolution Strategies can find
successful policies in Atari that outperform the policies found by Deep
Q-Learning. Evolution Strategies also outperform Augmented Random Search in
most benchmarks, demonstrating superior sample efficiency and robustness in
training linear policy networks.
