  The state space in Multiagent Reinforcement Learning (MARL) grows
exponentially with the agent number. Such a curse of dimensionality results in
poor scalability and low sample efficiency, inhibiting MARL for decades. To
break this curse, we propose a unified agent permutation framework that
exploits the permutation invariance (PI) and permutation equivariance (PE)
inductive biases to reduce the multiagent state space. Our insight is that
permuting the order of entities in the factored multiagent state space does not
change the information. Specifically, we propose two novel implementations: a
Dynamic Permutation Network (DPN) and a Hyper Policy Network (HPN). The core
idea is to build separate entity-wise PI input and PE output network modules to
connect the entity-factored state space and action space in an end-to-end way.
DPN achieves such connections by two separate module selection networks, which
consistently assign the same input module to the same input entity (guarantee
PI) and assign the same output module to the same entity-related output
(guarantee PE). To enhance the representation capability, HPN replaces the
module selection networks of DPN with hypernetworks to directly generate the
corresponding module weights. Extensive experiments in SMAC, Google Research
Football and MPE validate that the proposed methods significantly boost the
performance and the learning efficiency of existing MARL algorithms.
Remarkably, in SMAC, we achieve 100% win rates in almost all hard and
super-hard scenarios (never achieved before).
