  Recent progress in artificial intelligence through reinforcement learning
(RL) has shown great success on increasingly complex single-agent environments
and two-player turn-based games. However, the real-world contains multiple
agents, each learning and acting independently to cooperate and compete with
other agents, and environments reflecting this degree of complexity remain an
open challenge. In this work, we demonstrate for the first time that an agent
can achieve human-level in a popular 3D multiplayer first-person video game,
Quake III Arena Capture the Flag, using only pixels and game points as input.
These results were achieved by a novel two-tier optimisation process in which a
population of independent RL agents are trained concurrently from thousands of
parallel matches with agents playing in teams together and against each other
on randomly generated environments. Each agent in the population learns its own
internal reward signal to complement the sparse delayed reward from winning,
and selects actions using a novel temporally hierarchical representation that
enables the agent to reason at multiple timescales. During game-play, these
agents display human-like behaviours such as navigating, following, and
defending based on a rich learned representation that is shown to encode
high-level game knowledge. In an extensive tournament-style evaluation the
trained agents exceeded the win-rate of strong human players both as teammates
and opponents, and proved far stronger than existing state-of-the-art agents.
These results demonstrate a significant jump in the capabilities of artificial
agents, bringing us closer to the goal of human-level intelligence.
