  Reinforcement learning (RL) agents are particularly hard to train when
rewards are sparse. One common solution is to use intrinsic rewards to
encourage agents to explore their environment. However, recent intrinsic
exploration methods often use state-based novelty measures which reward
low-level exploration and may not scale to domains requiring more abstract
skills. Instead, we explore natural language as a general medium for
highlighting relevant abstractions in an environment. Unlike previous work, we
evaluate whether language can improve over existing exploration methods by
directly extending (and comparing to) competitive intrinsic exploration
baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These
language-based variants outperform their non-linguistic forms by 47-85% across
13 challenging tasks from the MiniGrid and MiniHack environment suites.
