  Multi-agent proximal policy optimization (MAPPO) has recently demonstrated
state-of-the-art performance on challenging multi-agent reinforcement learning
tasks. However, MAPPO still struggles with the credit assignment problem,
wherein the sheer difficulty in ascribing credit to individual agents' actions
scales poorly with team size. In this paper, we propose a multi-agent
reinforcement learning algorithm that adapts recent developments in credit
assignment to improve upon MAPPO. Our approach leverages partial reward
decoupling (PRD), which uses a learned attention mechanism to estimate which of
a particular agent's teammates are relevant to its learning updates. We use
this estimate to dynamically decompose large groups of agents into smaller,
more manageable subgroups. We empirically demonstrate that our approach,
PRD-MAPPO, decouples agents from teammates that do not influence their expected
future reward, thereby streamlining credit assignment. We additionally show
that PRD-MAPPO yields significantly higher data efficiency and asymptotic
performance compared to both MAPPO and other state-of-the-art methods across
several multi-agent tasks, including StarCraft II. Finally, we propose a
version of PRD-MAPPO that is applicable to \textit{shared} reward settings,
where PRD was previously not applicable, and empirically show that this also
leads to performance improvements over MAPPO.
