  Hierarchical Imitation Learning (HIL) has been proposed to recover
highly-complex behaviors in long-horizon tasks from expert demonstrations by
modeling the task hierarchy with the option framework. Existing methods either
overlook the causal relationship between the subtask and its corresponding
policy or cannot learn the policy in an end-to-end fashion, which leads to
suboptimality. In this work, we develop a novel HIL algorithm based on
Adversarial Inverse Reinforcement Learning and adapt it with the
Expectation-Maximization algorithm in order to directly recover a hierarchical
policy from the unannotated demonstrations. Further, we introduce a directed
information term to the objective function to enhance the causality and propose
a Variational Autoencoder framework for learning with our objectives in an
end-to-end fashion. Theoretical justifications and evaluations on challenging
robotic control tasks are provided to show the superiority of our algorithm.
The codes are available at https://github.com/LucasCJYSDL/HierAIRL.
