  Evolutionary reinforcement learning (ERL) algorithms recently raise attention
in tackling complex reinforcement learning (RL) problems due to high
parallelism, while they are prone to insufficient exploration or model collapse
without carefully tuning hyperparameters (aka meta-parameters). In the paper,
we propose a general meta ERL framework via bilevel optimization (BiERL) to
jointly update hyperparameters in parallel to training the ERL model within a
single agent, which relieves the need for prior domain knowledge or costly
optimization procedure before model deployment. We design an elegant meta-level
architecture that embeds the inner-level's evolving experience into an
informative population representation and introduce a simple and feasible
evaluation of the meta-level fitness function to facilitate learning
efficiency. We perform extensive experiments in MuJoCo and Box2D tasks to
verify that as a general framework, BiERL outperforms various baselines and
consistently improves the learning performance for a diversity of ERL
algorithms.
