  Hierarchical Reinforcement Learning (HRL) exploits temporal abstraction to
solve large Markov Decision Processes (MDP) and provide transferable subtask
policies. In this paper, we introduce an off-policy HRL algorithm: Hierarchical
Q-value Iteration (HQI). We show that it is possible to effectively learn
recursive optimal policies for any valid hierarchical decomposition of the
original MDP, given a fixed dataset collected from a flat stochastic behavioral
policy. We first formally prove the convergence of the algorithm for tabular
MDP. Then our experiments on the Taxi domain show that HQI converges faster
than a flat Q-value Iteration and enjoys easy state abstraction. Also, we
demonstrate that our algorithm is able to learn optimal policies for different
hierarchical structures from the same fixed dataset, which enables model
comparison without recollecting data.
