  The past decade has seen vast progress in deep reinforcement learning (RL) on
the back of algorithms manually designed by human researchers. Recently, it has
been shown that it is possible to meta-learn update rules, with the hope of
discovering algorithms that can perform well on a wide range of RL tasks.
Despite impressive initial results from algorithms such as Learned Policy
Gradient (LPG), there remains a generalization gap when these algorithms are
applied to unseen environments. In this work, we examine how characteristics of
the meta-training distribution impact the generalization performance of these
algorithms. Motivated by this analysis and building on ideas from Unsupervised
Environment Design (UED), we propose a novel approach for automatically
generating curricula to maximize the regret of a meta-learned optimizer, in
addition to a novel approximation of regret, which we name algorithmic regret
(AR). The result is our method, General RL Optimizers Obtained Via Environment
Design (GROOVE). In a series of experiments, we show that GROOVE achieves
superior generalization to LPG, and evaluate AR against baseline metrics from
UED, identifying it as a critical component of environment design in this
setting. We believe this approach is a step towards the discovery of truly
general RL algorithms, capable of solving a wide range of real-world
environments.
