  Despite impressive successes, deep reinforcement learning (RL) systems still
fall short of human performance on generalization to new tasks and environments
that differ from their training. As a benchmark tailored for studying RL
generalization, we introduce Avalon, a set of tasks in which embodied agents in
highly diverse procedural 3D worlds must survive by navigating terrain, hunting
or gathering food, and avoiding hazards. Avalon is unique among existing RL
benchmarks in that the reward function, world dynamics, and action space are
the same for every task, with tasks differentiated solely by altering the
environment; its 20 tasks, ranging in complexity from eat and throw to hunt and
navigate, each create worlds in which the agent must perform specific skills in
order to survive. This setup enables investigations of generalization within
tasks, between tasks, and to compositional tasks that require combining skills
learned from previous tasks. Avalon includes a highly efficient simulator, a
library of baselines, and a benchmark with scoring metrics evaluated against
hundreds of hours of human performance, all of which are open-source and
publicly available. We find that standard RL baselines make progress on most
tasks but are still far from human performance, suggesting Avalon is
challenging enough to advance the quest for generalizable RL.
