  Centralized Training with Decentralized Execution (CTDE) has been a popular
paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and
is widely used in many real applications. One of the major challenges in the
training process is credit assignment, which aims to deduce the contributions
of each agent according to the global rewards. Existing credit assignment
methods focus on either decomposing the joint value function into individual
value functions or measuring the impact of local observations and actions on
the global value function. These approaches lack a thorough consideration of
the complicated interactions among multiple agents, leading to an unsuitable
assignment of credit and subsequently mediocre results on MARL. We propose
Shapley Counterfactual Credit Assignment, a novel method for explicit credit
assignment which accounts for the coalition of agents. Specifically, Shapley
Value and its desired properties are leveraged in deep MARL to credit any
combinations of agents, which grants us the capability to estimate the
individual credit for each agent. Despite this capability, the main technical
difficulty lies in the computational complexity of Shapley Value who grows
factorially as the number of agents. We instead utilize an approximation method
via Monte Carlo sampling, which reduces the sample complexity while maintaining
its effectiveness. We evaluate our method on StarCraft II benchmarks across
different scenarios. Our method outperforms existing cooperative MARL
algorithms significantly and achieves the state-of-the-art, with especially
large margins on tasks with more severe difficulties.
