  Entropy regularization is a popular method in reinforcement learning (RL).
Although it has many advantages, it alters the RL objective of the original
Markov Decision Process (MDP). Though divergence regularization has been
proposed to settle this problem, it cannot be trivially applied to cooperative
multi-agent reinforcement learning (MARL). In this paper, we investigate
divergence regularization in cooperative MARL and propose a novel off-policy
cooperative MARL framework, divergence-regularized multi-agent actor-critic
(DMAC). Theoretically, we derive the update rule of DMAC which is naturally
off-policy and guarantees monotonic policy improvement and convergence in both
the original MDP and divergence-regularized MDP. We also give a bound of the
discrepancy between the converged policy and optimal policy in the original
MDP. DMAC is a flexible framework and can be combined with many existing MARL
algorithms. Empirically, we evaluate DMAC in a didactic stochastic game and
StarCraft Multi-Agent Challenge and show that DMAC substantially improves the
performance of existing MARL algorithms.
