  Deep reinforcement learning (DRL) algorithms and evolution strategies (ES)
have been applied to various tasks, showing excellent performances. These have
the opposite properties, with DRL having good sample efficiency and poor
stability, while ES being vice versa. Recently, there have been attempts to
combine these algorithms, but these methods fully rely on synchronous update
scheme, making it not ideal to maximize the benefits of the parallelism in ES.
To solve this challenge, asynchronous update scheme was introduced, which is
capable of good time-efficiency and diverse policy exploration. In this paper,
we introduce an Asynchronous Evolution Strategy-Reinforcement Learning (AES-RL)
that maximizes the parallel efficiency of ES and integrates it with policy
gradient methods. Specifically, we propose 1) a novel framework to merge ES and
DRL asynchronously and 2) various asynchronous update methods that can take all
advantages of asynchronism, ES, and DRL, which are exploration and time
efficiency, stability, and sample efficiency, respectively. The proposed
framework and update methods are evaluated in continuous control benchmark
work, showing superior performance as well as time efficiency compared to the
previous methods.
