  Multi-Agent Policy Gradient (MAPG) has made significant progress in recent
years. However, centralized critics in state-of-the-art MAPG methods still face
the centralized-decentralized mismatch (CDM) issue, which means sub-optimal
actions by some agents will affect other agent's policy learning. While using
individual critics for policy updates can avoid this issue, they severely limit
cooperation among agents. To address this issue, we propose an agent topology
framework, which decides whether other agents should be considered in policy
gradient and achieves compromise between facilitating cooperation and
alleviating the CDM issue. The agent topology allows agents to use coalition
utility as learning objective instead of global utility by centralized critics
or local utility by individual critics. To constitute the agent topology,
various models are studied. We propose Topology-based multi-Agent Policy
gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove
the policy improvement theorem for stochastic TAPE and give a theoretical
explanation for the improved cooperation among agents. Experiment results on
several benchmarks show the agent topology is able to facilitate agent
cooperation and alleviate CDM issue respectively to improve performance of
TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm
are devised to show the efficacy of the agent topology.
