  Solving tasks with sparse rewards is a main challenge in reinforcement
learning. While hierarchical controllers are an intuitive approach to this
problem, current methods often require manual reward shaping, alternating
training phases, or manually defined sub tasks. We introduce modulated policy
hierarchies (MPH), that can learn end-to-end to solve tasks from sparse
rewards. To achieve this, we study different modulation signals and exploration
for hierarchical controllers. Specifically, we find that communicating via
bit-vectors is more efficient than selecting one out of multiple skills, as it
enables mixing between them. To facilitate exploration, MPH uses its different
time scales for temporally extended intrinsic motivation at each level of the
hierarchy. We evaluate MPH on the robotics tasks of pushing and sparse block
stacking, where it outperforms recent baselines.
