  Multiagent reinforcement learning (MARL) can solve complex cooperative tasks.
However, the efficiency of existing MARL methods relies heavily on well-defined
reward functions. Multiagent tasks with sparse reward feedback are especially
challenging not only because of the credit distribution problem, but also due
to the low probability of obtaining positive reward feedback. In this paper, we
design a graph network called Cooperation Graph (CG). The Cooperation Graph is
the combination of two simple bipartite graphs, namely, the Agent Clustering
subgraph (ACG) and the Cluster Designating subgraph (CDG). Next, based on this
novel graph structure, we propose a Cooperation Graph Multiagent Reinforcement
Learning (CG-MARL) algorithm, which can efficiently deal with the sparse reward
problem in multiagent tasks. In CG-MARL, agents are directly controlled by the
Cooperation Graph. And a policy neural network is trained to manipulate this
Cooperation Graph, guiding agents to achieve cooperation in an implicit way.
This hierarchical feature of CG-MARL provides space for customized
cluster-actions, an extensible interface for introducing fundamental
cooperation knowledge. In experiments, CG-MARL shows state-of-the-art
performance in sparse reward multiagent benchmarks, including the anti-invasion
interception task and the multi-cargo delivery task.
