  Many tasks in artificial intelligence require the collaboration of multiple
agents. We exam deep reinforcement learning for multi-agent domains. Recent
research efforts often take the form of two seemingly conflicting perspectives,
the decentralized perspective, where each agent is supposed to have its own
controller; and the centralized perspective, where one assumes there is a
larger model controlling all agents. In this regard, we revisit the idea of the
master-slave architecture by incorporating both perspectives within one
framework. Such a hierarchical structure naturally leverages advantages from
one another. The idea of combining both perspectives is intuitive and can be
well motivated from many real world systems, however, out of a variety of
possible realizations, we highlights three key ingredients, i.e. composed
action representation, learnable communication and independent reasoning. With
network designs to facilitate these explicitly, our proposal consistently
outperforms latest competing methods both in synthetic experiments and when
applied to challenging StarCraft micromanagement tasks.
