  Multi-agent reinforcement learning (MARL) methods often suffer from high
sample complexity, limiting their use in real-world problems where data is
sparse or expensive to collect. Although latent-variable world models have been
employed to address this issue by generating abundant synthetic data for MARL
training, most of these models cannot encode vital global information available
during training into their latent states, which hampers learning efficiency.
The few exceptions that incorporate global information assume centralized
execution of their learned policies, which is impractical in many applications
with partial observability.
  We propose a novel model-based MARL algorithm, MABL (Multi-Agent Bi-Level
world model), that learns a bi-level latent-variable world model from
high-dimensional inputs. Unlike existing models, MABL is capable of encoding
essential global information into the latent states during training while
guaranteeing the decentralized execution of learned policies. For each agent,
MABL learns a global latent state at the upper level, which is used to inform
the learning of an agent latent state at the lower level. During execution,
agents exclusively use lower-level latent states and act independently.
Crucially, MABL can be combined with any model-free MARL algorithm for policy
learning. In our empirical evaluation with complex discrete and continuous
multi-agent tasks including SMAC, Flatland, and MAMuJoCo, MABL surpasses SOTA
multi-agent latent-variable world models in both sample efficiency and overall
performance.
