  Reinforcement learning practitioners often avoid hierarchical policies,
especially in image-based observation spaces. Typically, the single-task
performance improvement over flat-policy counterparts does not justify the
additional complexity associated with implementing a hierarchy. However, by
introducing multiple decision-making levels, hierarchical policies can compose
lower-level policies to more effectively generalize between tasks, highlighting
the need for multi-task evaluations. We analyze the benefits of hierarchy
through simulated multi-task robotic control experiments from pixels. Our
results show that hierarchical policies trained with task conditioning can (1)
increase performance on training tasks, (2) lead to improved reward and
state-space generalizations in similar tasks, and (3) decrease the complexity
of fine tuning required to solve novel tasks. Thus, we believe that
hierarchical policies should be considered when building reinforcement learning
architectures capable of generalizing between tasks.
