  The rise of multi-agent systems, especially the success of multi-agent
reinforcement learning (MARL), is reshaping our future across diverse domains
like autonomous vehicle networks. However, MARL still faces significant
challenges, particularly in achieving zero-shot scalability, which allows
trained MARL models to be directly applied to unseen tasks with varying numbers
of agents. In addition, real-world multi-agent systems usually contain agents
with different functions and strategies, while the existing scalable MARL
methods only have limited heterogeneity. To address this, we propose a novel
MARL framework named Scalable and Heterogeneous Proximal Policy Optimization
(SHPPO), integrating heterogeneity into parameter-shared PPO-based MARL
networks. we first leverage a latent network to adaptively learn strategy
patterns for each agent. Second, we introduce a heterogeneous layer for
decision-making, whose parameters are specifically generated by the learned
latent variables. Our approach is scalable as all the parameters are shared
except for the heterogeneous layer, and gains both inter-individual and
temporal heterogeneity at the same time. We implement our approach based on the
state-of-the-art backbone PPO-based algorithm as SHPPO, while our approach is
agnostic to the backbone and can be seamlessly plugged into any
parameter-shared MARL method. SHPPO exhibits superior performance over the
baselines such as MAPPO and HAPPO in classic MARL environments like Starcraft
Multi-Agent Challenge (SMAC) and Google Research Football (GRF), showcasing
enhanced zero-shot scalability and offering insights into the learned latent
representation's impact on team performance by visualization.
