  Large-scale heterogeneous multiagent systems feature various realistic
factors in the real world, such as agents with diverse abilities and overall
system cost. In comparison to homogeneous systems, heterogeneous systems offer
significant practical advantages. Nonetheless, they also present challenges for
multiagent reinforcement learning, including addressing the non-stationary
problem and managing an imbalanced number of agents with different types. We
propose a Prioritized Heterogeneous League Reinforcement Learning (PHLRL)
method to address large-scale heterogeneous cooperation problems. PHLRL
maintains a record of various policies that agents have explored during their
training and establishes a heterogeneous league consisting of diverse policies
to aid in future policy optimization. Furthermore, we design a prioritized
policy gradient approach to compensate for the gap caused by differences in the
number of different types of agents. Next, we use Unreal Engine to design a
large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent
Operation (LSMO), which is a complex two-team competition scenario that
requires collaboration from both ground and airborne agents. We use experiments
to show that PHLRL outperforms state-of-the-art methods, including QTRAN and
QPLEX in LSMO.
