  Intelligent behaviour in the physical world exhibits structure at multiple
spatial and temporal scales. Although movements are ultimately executed at the
level of instantaneous muscle tensions or joint torques, they must be selected
to serve goals defined on much longer timescales, and in terms of relations
that extend far beyond the body itself, ultimately involving coordination with
other agents. Recent research in artificial intelligence has shown the promise
of learning-based approaches to the respective problems of complex movement,
longer-term planning and multi-agent coordination. However, there is limited
research aimed at their integration. We study this problem by training teams of
physically simulated humanoid avatars to play football in a realistic virtual
environment. We develop a method that combines imitation learning, single- and
multi-agent reinforcement learning and population-based training, and makes use
of transferable representations of behaviour for decision making at different
levels of abstraction. In a sequence of stages, players first learn to control
a fully articulated body to perform realistic, human-like movements such as
running and turning; they then acquire mid-level football skills such as
dribbling and shooting; finally, they develop awareness of others and play as a
team, bridging the gap between low-level motor control at a timescale of
milliseconds, and coordinated goal-directed behaviour as a team at the
timescale of tens of seconds. We investigate the emergence of behaviours at
different levels of abstraction, as well as the representations that underlie
these behaviours using several analysis techniques, including statistics from
real-world sports analytics. Our work constitutes a complete demonstration of
integrated decision-making at multiple scales in a physically embodied
multi-agent setting. See project video at https://youtu.be/KHMwq9pv7mg.
