  Reinforcement Learning (RL) has achieved impressive performance in many
complex environments due to the integration with Deep Neural Networks (DNNs).
At the same time, Genetic Algorithms (GAs), often seen as a competing approach
to RL, had limited success in scaling up to the DNNs required to solve
challenging tasks. Contrary to this dichotomic view, in the physical world,
evolution and learning are complementary processes that continuously interact.
The recently proposed Evolutionary Reinforcement Learning (ERL) framework has
demonstrated mutual benefits to performance when combining the two methods.
However, ERL has not fully addressed the scalability problem of GAs. In this
paper, we show that this problem is rooted in an unfortunate combination of a
simple genetic encoding for DNNs and the use of traditional
biologically-inspired variation operators. When applied to these encodings, the
standard operators are destructive and cause catastrophic forgetting of the
traits the networks acquired. We propose a novel algorithm called Proximal
Distilled Evolutionary Reinforcement Learning (PDERL) that is characterised by
a hierarchical integration between evolution and learning. The main innovation
of PDERL is the use of learning-based variation operators that compensate for
the simplicity of the genetic representation. Unlike traditional operators, our
proposals meet the functional requirements of variation operators when applied
on directly-encoded DNNs. We evaluate PDERL in five robot locomotion settings
from the OpenAI gym. Our method outperforms ERL, as well as two
state-of-the-art RL algorithms, PPO and TD3, in all tested environments.
