  Learning to collaborate is critical in Multi-Agent Reinforcement Learning
(MARL). Previous works promote collaboration by maximizing the correlation of
agents' behaviors, which is typically characterized by Mutual Information (MI)
in different forms. However, we reveal sub-optimal collaborative behaviors also
emerge with strong correlations, and simply maximizing the MI can,
surprisingly, hinder the learning towards better collaboration. To address this
issue, we propose a novel MARL framework, called Progressive Mutual Information
Collaboration (PMIC), for more effective MI-driven collaboration. PMIC uses a
new collaboration criterion measured by the MI between global states and joint
actions. Based on this criterion, the key idea of PMIC is maximizing the MI
associated with superior collaborative behaviors and minimizing the MI
associated with inferior ones. The two MI objectives play complementary roles
by facilitating better collaborations while avoiding falling into sub-optimal
ones. Experiments on a wide range of MARL benchmarks show the superior
performance of PMIC compared with other algorithms.
