  Consider the problem of exploration in sparse-reward or reward-free
environments, such as in Montezuma's Revenge. In the curiosity-driven paradigm,
the agent is rewarded for how much each realized outcome differs from their
predicted outcome. But using predictive error as intrinsic motivation is
fragile in stochastic environments, as the agent may become trapped by
high-entropy areas of the state-action space, such as a "noisy TV". In this
work, we study a natural solution derived from structural causal models of the
world: Our key idea is to learn representations of the future that capture
precisely the unpredictable aspects of each outcome -- which we use as
additional input for predictions, such that intrinsic rewards only reflect the
predictable aspects of world dynamics. First, we propose incorporating such
hindsight representations into models to disentangle "noise" from "novelty",
yielding Curiosity in Hindsight: a simple and scalable generalization of
curiosity that is robust to stochasticity. Second, we instantiate this
framework for the recently introduced BYOL-Explore algorithm as our prime
example, resulting in the noise-robust BYOL-Hindsight. Third, we illustrate its
behavior under a variety of different stochasticities in a grid world, and find
improvements over BYOL-Explore in hard-exploration Atari games with sticky
actions. Notably, we show state-of-the-art results in exploring Montezuma's
Revenge with sticky actions, while preserving performance in the non-sticky
setting.
