  Deep reinforcement learning methods have shown great performance on many
challenging cooperative multi-agent tasks. Two main promising research
directions are multi-agent value function decomposition and multi-agent policy
gradients. In this paper, we propose a new decomposed multi-agent soft
actor-critic (mSAC) method, which effectively combines the advantages of the
aforementioned two methods. The main modules include decomposed Q network
architecture, discrete probabilistic policy and counterfactual advantage
function (optinal). Theoretically, mSAC supports efficient off-policy learning
and addresses credit assignment problem partially in both discrete and
continuous action spaces. Tested on StarCraft II micromanagement cooperative
multiagent benchmark, we empirically investigate the performance of mSAC
against its variants and analyze the effects of the different components.
Experimental results demonstrate that mSAC significantly outperforms
policy-based approach COMA, and achieves competitive results with SOTA
value-based approach Qmix on most tasks in terms of asymptotic perfomance
metric. In addition, mSAC achieves pretty good results on large action space
tasks, such as 2c_vs_64zg and MMM2.
