  In recent years, a number of reinforcement learning (RL) methods have been
proposed to explore complex environments which differ across episodes. In this
work, we show that the effectiveness of these methods critically relies on a
count-based episodic term in their exploration bonus. As a result, despite
their success in relatively simple, noise-free settings, these methods fall
short in more realistic scenarios where the state space is vast and prone to
noise. To address this limitation, we introduce Exploration via Elliptical
Episodic Bonuses (E3B), a new method which extends count-based episodic bonuses
to continuous state spaces and encourages an agent to explore states that are
diverse under a learned embedding within each episode. The embedding is learned
using an inverse dynamics model in order to capture controllable aspects of the
environment. Our method sets a new state-of-the-art across 16 challenging tasks
from the MiniHack suite, without requiring task-specific inductive biases. E3B
also matches existing methods on sparse reward, pixel-based VizDoom
environments, and outperforms existing methods in reward-free exploration on
Habitat, demonstrating that it can scale to high-dimensional pixel-based
observations and realistic environments.
