  Many multiagent systems in the real world include multiple types of agents
with different abilities and functionality. Such heterogeneous multiagent
systems have significant practical advantages. However, they also come with
challenges compared with homogeneous systems for multiagent reinforcement
learning, such as the non-stationary problem and the policy version iteration
issue. This work proposes a general-purpose reinforcement learning algorithm
named Heterogeneous League Training (HLT) to address heterogeneous multiagent
problems. HLT keeps track of a pool of policies that agents have explored
during training, gathering a league of heterogeneous policies to facilitate
future policy optimization. Moreover, a hyper-network is introduced to increase
the diversity of agent behaviors when collaborating with teammates having
different levels of cooperation skills. We use heterogeneous benchmark tasks to
demonstrate that (1) HLT promotes the success rate in cooperative heterogeneous
tasks; (2) HLT is an effective approach to solving the policy version iteration
problem; (3) HLT provides a practical way to assess the difficulty of learning
each role in a heterogeneous team.
