  Modern deep reinforcement learning (RL) algorithms are motivated by either
the generalised policy iteration (GPI) or trust-region learning (TRL)
frameworks. However, algorithms that strictly respect these theoretical
frameworks have proven unscalable. Surprisingly, the only known scalable
algorithms violate the GPI/TRL assumptions, e.g. due to required regularisation
or other heuristics. The current explanation of their empirical success is
essentially "by analogy": they are deemed approximate adaptations of
theoretically sound methods. Unfortunately, studies have shown that in practice
these algorithms differ greatly from their conceptual ancestors. In contrast,
in this paper we introduce a novel theoretical framework, named Mirror
Learning, which provides theoretical guarantees to a large class of algorithms,
including TRPO and PPO. While the latter two exploit the flexibility of our
framework, GPI and TRL fit in merely as pathologically restrictive corner cases
thereof. This suggests that the empirical performance of state-of-the-art
methods is a direct consequence of their theoretical properties, rather than of
aforementioned approximate analogies. Mirror learning sets us free to boldly
explore novel, theoretically sound RL algorithms, a thus far uncharted
wonderland.
