  Exploration bonuses in reinforcement learning guide long-horizon exploration
by defining custom intrinsic objectives. Several exploration objectives like
count-based bonuses, pseudo-counts, and state-entropy maximization are
non-stationary and hence are difficult to optimize for the agent. While this
issue is generally known, it is usually omitted and solutions remain
under-explored. The key contribution of our work lies in transforming the
original non-stationary rewards into stationary rewards through an augmented
state representation. For this purpose, we introduce the Stationary Objectives
For Exploration (SOFE) framework. SOFE requires identifying sufficient
statistics for different exploration bonuses and finding an efficient encoding
of these statistics to use as input to a deep network. SOFE is based on
proposing state augmentations that expand the state space but hold the promise
of simplifying the optimization of the agent's objective. We show that SOFE
improves the performance of several exploration objectives, including
count-based bonuses, pseudo-counts, and state-entropy maximization. Moreover,
SOFE outperforms prior methods that attempt to stabilize the optimization of
intrinsic objectives. We demonstrate the efficacy of SOFE in hard-exploration
problems, including sparse-reward tasks, pixel-based observations, 3D
navigation, and procedurally generated environments.
