HAC Explore: Accelerating Exploration with Hierarchical Reinforcement
  Learning
  Sparse rewards and long time horizons remain challenging for reinforcement
learning algorithms. Exploration bonuses can help in sparse reward settings by
encouraging agents to explore the state space, while hierarchical approaches
can assist with long-horizon tasks by decomposing lengthy tasks into shorter
subtasks. We propose HAC Explore (HACx), a new method that combines these
approaches by integrating the exploration bonus method Random Network
Distillation (RND) into the hierarchical approach Hierarchical Actor-Critic
(HAC). HACx outperforms either component method on its own, as well as an
existing approach to combining hierarchy and exploration, in a set of difficult
simulated robotics tasks. HACx is the first RL method to solve a sparse reward,
continuous-control task that requires over 1,000 actions.
