  Recently, some challenging tasks in multi-agent systems have been solved by
some hierarchical reinforcement learning methods. Inspired by the intra-level
and inter-level coordination in the human nervous system, we propose a novel
value decomposition framework HAVEN based on hierarchical reinforcement
learning for fully cooperative multi-agent problems. To address the instability
arising from the concurrent optimization of policies between various levels and
agents, we introduce the dual coordination mechanism of inter-level and
inter-agent strategies by designing reward functions in a two-level hierarchy.
HAVEN does not require domain knowledge and pre-training, and can be applied to
any value decomposition variant. Our method achieves desirable results on
different decentralized partially observable Markov decision process domains
and outperforms other popular multi-agent hierarchical reinforcement learning
algorithms.
