  Building agents capable of understanding language instructions is critical to
effective and robust human-AI collaboration. Recent work focuses on training
these agents via reinforcement learning in environments with synthetic
language; however, instructions often define long-horizon, sparse-reward tasks,
and learning policies requires many episodes of experience. We introduce ELLA:
Exploration through Learned Language Abstraction, a reward shaping approach
geared towards boosting sample efficiency in sparse reward environments by
correlating high-level instructions with simpler low-level constituents. ELLA
has two key elements: 1) A termination classifier that identifies when agents
complete low-level instructions, and 2) A relevance classifier that correlates
low-level instructions with success on high-level tasks. We learn the
termination classifier offline from pairs of instructions and terminal states.
Notably, in departure from prior work in language and abstraction, we learn the
relevance classifier online, without relying on an explicit decomposition of
high-level instructions to low-level instructions. On a suite of complex BabyAI
environments with varying instruction complexities and reward sparsity, ELLA
shows gains in sample efficiency relative to language-based shaping and
traditional RL methods.
