  Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a
promising approach to solving long-horizon tasks. Recently, its success has
been extended to more general settings by concurrently learning hierarchical
policies and subgoal representations. Although GCHRL possesses superior
exploration ability by decomposing tasks via subgoals, existing GCHRL methods
struggle in temporally extended tasks with sparse external rewards, since the
high-level policy learning relies on external rewards. As the high-level policy
selects subgoals in an online learned representation space, the dynamic change
of the subgoal space severely hinders effective high-level exploration. In this
paper, we propose a novel regularization that contributes to both stable and
efficient subgoal representation learning. Building upon the stable
representation, we design measures of novelty and potential for subgoals, and
develop an active hierarchical exploration strategy that seeks out new
promising subgoals and states without intrinsic rewards. Experimental results
show that our approach significantly outperforms state-of-the-art baselines in
continuous control tasks with sparse rewards.
