  Evolutionary Reinforcement Learning (ERL) that applying Evolutionary
Algorithms (EAs) to optimize the weight parameters of Deep Neural Network (DNN)
based policies has been widely regarded as an alternative to traditional
reinforcement learning methods. However, the evaluation of the iteratively
generated population usually requires a large amount of computational time and
can be prohibitively expensive, which may potentially restrict the
applicability of ERL. Surrogate is often used to reduce the computational
burden of evaluation in EAs. Unfortunately, in ERL, each individual of policy
usually represents millions of weights parameters of DNN. This high-dimensional
representation of policy has introduced a great challenge to the application of
surrogates into ERL to speed up training. This paper proposes a PE-SAERL
Framework to at the first time enable surrogate-assisted evolutionary
reinforcement learning via policy embedding (PE). Empirical results on 5 Atari
games show that the proposed method can perform more efficiently than the four
state-of-the-art algorithms. The training process is accelerated up to 7x on
tested games, comparing to its counterpart without the surrogate and PE.
