  For an intelligent agent to flexibly and efficiently operate in complex
environments, they must be able to reason at multiple levels of temporal,
spatial, and conceptual abstraction. At the lower levels, the agent must
interpret their proprioceptive inputs and control their muscles, and at the
higher levels, the agent must select goals and plan how they will achieve those
goals. It is clear that each of these types of reasoning is amenable to
different types of representations, algorithms, and inputs. In this work, we
introduce a simple, three-level hierarchical architecture that reflects these
distinctions. The low-level controller operates on the continuous
proprioceptive inputs, using model-free learning to acquire useful behaviors.
These in turn induce a set of mid-level dynamics, which are learned by the
mid-level controller and used for model-predictive control, to select a
behavior to activate at each timestep. The high-level controller leverages a
discrete, graph representation for goal selection and path planning to specify
targets for the mid-level controller. We apply our method to a series of
navigation tasks in the Mujoco Ant environment, consistently demonstrating
significant improvements in sample-efficiency compared to prior model-free,
model-based, and hierarchical RL methods. Finally, as an illustrative example
of the advantages of our architecture, we apply our method to a complex maze
environment that requires efficient exploration and long-horizon planning.
