  In this paper we analyze the qualitative differences between evolutionary
strategies and reinforcement learning algorithms by focusing on two popular
state-of-the-art algorithms: the OpenAI-ES evolutionary strategy and the
Proximal Policy Optimization (PPO) reinforcement learning algorithm -- the most
similar methods of the two families. We analyze how the methods differ with
respect to: (i) general efficacy, (ii) ability to cope with sparse rewards,
(iii) propensity/capacity to discover minimal solutions, (iv) dependency on
reward shaping, and (v) ability to cope with variations of the environmental
conditions. The analysis of the performance and of the behavioral strategies
displayed by the agents trained with the two methods on benchmark problems
enable us to demonstrate qualitative differences which were not identified in
previous studies, to identify the relative weakness of the two methods, and to
propose ways to ameliorate some of those weakness. We show that the
characteristics of the reward function has a strong impact which vary
qualitatively not only for the OpenAI-ES and the PPO but also for alternative
reinforcement learning algorithms, thus demonstrating the importance of
optimizing the characteristic of the reward function to the algorithm used.
