  Recent successes in Reinforcement Learning have encouraged a fast-growing
network of RL researchers and a number of breakthroughs in RL research. As the
RL community and the body of RL work grows, so does the need for widely
applicable benchmarks that can fairly and effectively evaluate a variety of RL
algorithms.
  This need is particularly apparent in the realm of Hierarchical Reinforcement
Learning (HRL). While many existing test domains may exhibit hierarchical
action or state structures, modern RL algorithms still exhibit great difficulty
in solving domains that necessitate hierarchical modeling and action planning,
even when such domains are seemingly trivial. These difficulties highlight both
the need for more focus on HRL algorithms themselves, and the need for new
testbeds that will encourage and validate HRL research.
  Existing HRL testbeds exhibit a Goldilocks problem; they are often either too
simple (e.g. Taxi) or too complex (e.g. Montezuma's Revenge from the Arcade
Learning Environment). In this paper we present the Escape Room Domain (ERD), a
new flexible, scalable, and fully implemented testing domain for HRL that
bridges the "moderate complexity" gap left behind by existing alternatives. ERD
is open-source and freely available through GitHub, and conforms to widely-used
public testing interfaces for simple integration and testing with a variety of
public RL agent implementations. We show that the ERD presents a suite of
challenges with scalable difficulty to provide a smooth learning gradient from
Taxi to the Arcade Learning Environment.
