  Many tasks in practice require the collaboration of multiple agents through
reinforcement learning. In general, cooperative multiagent reinforcement
learning algorithms can be classified into two paradigms: Joint Action Learners
(JALs) and Independent Learners (ILs). In many practical applications, agents
are unable to observe other agents' actions and rewards, making JALs
inapplicable. In this work, we focus on independent learning paradigm in which
each agent makes decisions based on its local observations only. However,
learning is challenging in independent settings due to the local viewpoints of
all agents, which perceive the world as a non-stationary environment due to the
concurrently exploring teammates. In this paper, we propose a novel framework
called Independent Generative Adversarial Self-Imitation Learning (IGASIL) to
address the coordination problems in fully cooperative multiagent environments.
To the best of our knowledge, we are the first to combine self-imitation
learning with generative adversarial imitation learning (GAIL) and apply it to
cooperative multiagent systems. Besides, we put forward a Sub-Curriculum
Experience Replay mechanism to pick out the past beneficial experiences as much
as possible and accelerate the self-imitation learning process. Evaluations
conducted in the testbed of StarCraft unit micromanagement and a commonly
adopted benchmark show that our IGASIL produces state-of-the-art results and
even outperforms JALs in terms of both convergence speed and final performance.
