  One of the most challenging aspects of real-world reinforcement learning (RL)
is the multitude of unpredictable and ever-changing distractions that could
divert an agent from what was tasked to do in its training environment. While
an agent could learn from reward signals to ignore them, the complexity of the
real-world can make rewards hard to acquire, or, at best, extremely sparse. A
recent class of self-supervised methods have shown promise that reward-free
adaptation under challenging distractions is possible. However, previous work
focused on a short one-episode adaptation setting. In this paper, we consider a
long-term adaptation setup that is more akin to the specifics of the real-world
and propose a geometric perspective on self-supervised adaptation. We
empirically describe the processes that take place in the embedding space
during this adaptation process, reveal some of its undesirable effects on
performance and show how they can be eliminated. Moreover, we theoretically
study how actor-based and actor-free agents can further generalise to the
target environment by manipulating the geometry of the manifolds described by
the actor and critic functions.
