  The option framework has shown great promise by automatically extracting
temporally-extended sub-tasks from a long-horizon task. Methods have been
proposed for concurrently learning low-level intra-option policies and
high-level option selection policy. However, existing methods typically suffer
from two major challenges: ineffective exploration and unstable updates. In
this paper, we present a novel and stable off-policy approach that builds on
the maximum entropy model to address these challenges. Our approach introduces
an information-theoretical intrinsic reward for encouraging the identification
of diverse and effective options. Meanwhile, we utilize a probability inference
model to simplify the optimization problem as fitting optimal trajectories.
Experimental results demonstrate that our approach significantly outperforms
prior on-policy and off-policy methods in a range of Mujoco benchmark tasks
while still providing benefits for transfer learning. In these tasks, our
approach learns a diverse set of options, each of whose state-action space has
strong coherence.
