  Envisioned application areas for reinforcement learning (RL) include
autonomous driving, precision agriculture, and finance, which all require RL
agents to make decisions in the real world. A significant challenge hindering
the adoption of RL methods in these domains is the non-robustness of
conventional algorithms. In this paper, we argue that a fundamental issue
contributing to this lack of robustness lies in the focus on the expected value
of the return as the sole ``correct'' optimization objective. The expected
value is the average over the statistical ensemble of infinitely many
trajectories. For non-ergodic returns, this average differs from the average
over a single but infinitely long trajectory. Consequently, optimizing the
expected value can lead to policies that yield exceptionally high returns with
probability zero but almost surely result in catastrophic outcomes. This
problem can be circumvented by transforming the time series of collected
returns into one with ergodic increments. This transformation enables learning
robust policies by optimizing the long-term return for individual agents rather
than the average across infinitely many trajectories. We propose an algorithm
for learning ergodicity transformations from data and demonstrate its
effectiveness in an instructive, non-ergodic environment and on standard RL
benchmarks.
