  Tremendous progress has been made in reinforcement learning (RL) over the
past decade. Most of these advancements came through the continual development
of new algorithms, which were designed using a combination of mathematical
derivations, intuitions, and experimentation. Such an approach of creating
algorithms manually is limited by human understanding and ingenuity. In
contrast, meta-learning provides a toolkit for automatic machine learning
method optimisation, potentially addressing this flaw. However, black-box
approaches which attempt to discover RL algorithms with minimal prior structure
have thus far not outperformed existing hand-crafted algorithms. Mirror
Learning, which includes RL algorithms, such as PPO, offers a potential
middle-ground starting point: while every method in this framework comes with
theoretical guarantees, components that differentiate them are subject to
design. In this paper we explore the Mirror Learning space by meta-learning a
"drift" function. We refer to the immediate result as Learnt Policy
Optimisation (LPO). By analysing LPO we gain original insights into policy
optimisation which we use to formulate a novel, closed-form RL algorithm,
Discovered Policy Optimisation (DPO). Our experiments in Brax environments
confirm state-of-the-art performance of LPO and DPO, as well as their transfer
to unseen settings.
