  Hierarchical DQN (h-DQN) is a two-level architecture of feedforward neural
networks where the meta level selects goals and the lower level takes actions
to achieve the goals. We show tasks that cannot be solved by h-DQN,
exemplifying the limitation of this type of hierarchical framework (HF). We
describe the recurrent hierarchical framework (RHF), generalizing architectures
that use a recurrent neural network at the meta level. We analyze the
expressiveness of HF and RHF using context-sensitive grammars. We show that RHF
is more expressive than HF. We perform experiments comparing an implementation
of RHF with two HF baselines; the results corroborate our theoretical findings.
