  Multi-agent football poses an unsolved challenge in AI research. Existing
work has focused on tackling simplified scenarios of the game, or else
leveraging expert demonstrations. In this paper, we develop a multi-agent
system to play the full 11 vs. 11 game mode, without demonstrations. This game
mode contains aspects that present major challenges to modern reinforcement
learning algorithms; multi-agent coordination, long-term planning, and
non-transitivity. To address these challenges, we present TiZero; a
self-evolving, multi-agent system that learns from scratch. TiZero introduces
several innovations, including adaptive curriculum learning, a novel self-play
strategy, and an objective that optimizes the policies of multiple agents
jointly. Experimentally, it outperforms previous systems by a large margin on
the Google Research Football environment, increasing win rates by over 30%. To
demonstrate the generality of TiZero's innovations, they are assessed on
several environments beyond football; Overcooked, Multi-agent
Particle-Environment, Tic-Tac-Toe and Connect-Four.
