  Deep Reinforcement Learning (DRL) algorithms have been successfully applied
to a range of challenging control tasks. However, these methods typically
suffer from three core difficulties: temporal credit assignment with sparse
rewards, lack of effective exploration, and brittle convergence properties that
are extremely sensitive to hyperparameters. Collectively, these challenges
severely limit the applicability of these approaches to real-world problems.
Evolutionary Algorithms (EAs), a class of black box optimization techniques
inspired by natural evolution, are well suited to address each of these three
challenges. However, EAs typically suffer from high sample complexity and
struggle to solve problems that require optimization of a large number of
parameters. In this paper, we introduce Evolutionary Reinforcement Learning
(ERL), a hybrid algorithm that leverages the population of an EA to provide
diversified data to train an RL agent, and reinserts the RL agent into the EA
population periodically to inject gradient information into the EA. ERL
inherits EA's ability of temporal credit assignment with a fitness metric,
effective exploration with a diverse set of policies, and stability of a
population-based approach and complements it with off-policy DRL's ability to
leverage gradients for higher sample efficiency and faster learning.
Experiments in a range of challenging continuous control benchmarks demonstrate
that ERL significantly outperforms prior DRL and EA methods.
