  In this paper, we propose a new mutual information framework for multi-agent
reinforcement learning to enable multiple agents to learn coordinated behaviors
by regularizing the accumulated return with the simultaneous mutual information
between multi-agent actions. By introducing a latent variable to induce nonzero
mutual information between multi-agent actions and applying a variational
bound, we derive a tractable lower bound on the considered MMI-regularized
objective function. The derived tractable objective can be interpreted as
maximum entropy reinforcement learning combined with uncertainty reduction of
other agents actions. Applying policy iteration to maximize the derived lower
bound, we propose a practical algorithm named variational maximum mutual
information multi-agent actor-critic, which follows centralized learning with
decentralized execution. We evaluated VM3-AC for several games requiring
coordination, and numerical results show that VM3-AC outperforms other MARL
algorithms in multi-agent tasks requiring high-quality coordination.
