  Multi-agent reinforcement learning (MARL) becomes more challenging in the
presence of more agents, as the capacity of the joint state and action spaces
grows exponentially in the number of agents. To address such a challenge of
scale, we identify a class of cooperative MARL problems with permutation
invariance, and formulate it as a mean-field Markov decision processes (MDP).
To exploit the permutation invariance therein, we propose the mean-field
proximal policy optimization (MF-PPO) algorithm, at the core of which is a
permutation-invariant actor-critic neural architecture. We prove that MF-PPO
attains the globally optimal policy at a sublinear rate of convergence.
Moreover, its sample complexity is independent of the number of agents. We
validate the theoretical advantages of MF-PPO with numerical experiments in the
multi-agent particle environment (MPE). In particular, we show that the
inductive bias introduced by the permutation-invariant neural architecture
enables MF-PPO to outperform existing competitors with a smaller number of
model parameters, which is the key to its generalization performance.
