  We address the problem of learning hierarchical deep neural network policies
for reinforcement learning. In contrast to methods that explicitly restrict or
cripple lower layers of a hierarchy to force them to use higher-level
modulating signals, each layer in our framework is trained to directly solve
the task, but acquires a range of diverse strategies via a maximum entropy
reinforcement learning objective. Each layer is also augmented with latent
random variables, which are sampled from a prior distribution during the
training of that layer. The maximum entropy objective causes these latent
variables to be incorporated into the layer's policy, and the higher level
layer can directly control the behavior of the lower layer through this latent
space. Furthermore, by constraining the mapping from latent variables to
actions to be invertible, higher layers retain full expressivity: neither the
higher layers nor the lower layers are constrained in their behavior. Our
experimental evaluation demonstrates that we can improve on the performance of
single-layer policies on standard benchmark tasks simply by adding additional
layers, and that our method can solve more complex sparse-reward tasks by
learning higher-level policies on top of high-entropy skills optimized for
simple low-level objectives.
