  Reinforcement learning can train policies that effectively perform complex
tasks. However for long-horizon tasks, the performance of these methods
degrades with horizon, often necessitating reasoning over and chaining
lower-level skills. Hierarchical reinforcement learning aims to enable this by
providing a bank of low-level skills as action abstractions. Hierarchies can
further improve on this by abstracting the space states as well. We posit that
a suitable state abstraction should depend on the capabilities of the available
lower-level policies. We propose Value Function Spaces: a simple approach that
produces such a representation by using the value functions corresponding to
each lower-level skill. These value functions capture the affordances of the
scene, thus forming a representation that compactly abstracts task relevant
information and robustly ignores distractors. Empirical evaluations for
maze-solving and robotic manipulation tasks demonstrate that our approach
improves long-horizon performance and enables better zero-shot generalization
than alternative model-free and model-based methods.
