  Many real-world applications involve some agents that fall into two teams,
with payoffs that are equal within the same team but of opposite sign across
the opponent team. The so-called two-team zero-sum Markov games (2t0sMGs) can
be resolved with reinforcement learning in recent years. However, existing
methods are thus inefficient in light of insufficient consideration of
intra-team credit assignment, data utilization and computational
intractability. In this paper, we propose the individual-global-minimax (IGMM)
principle to ensure the coherence between two-team minimax behaviors and the
individual greedy behaviors through Q functions in 2t0sMGs. Based on it, we
present a novel multi-agent reinforcement learning framework, Factorized
Multi-Agent MiniMax Q-Learning (FM3Q), which can factorize the joint minimax Q
function into individual ones and iteratively solve for the IGMM-satisfied
minimax Q functions for 2t0sMGs. Moreover, an online learning algorithm with
neural networks is proposed to implement FM3Q and obtain the deterministic and
decentralized minimax policies for two-team players. A theoretical analysis is
provided to prove the convergence of FM3Q. Empirically, we use three
environments to evaluate the learning efficiency and final performance of FM3Q
and show its superiority on 2t0sMGs.
