  Recent progress in the field of reinforcement learning has been accelerated
by virtual learning environments such as video games, where novel algorithms
and ideas can be quickly tested in a safe and reproducible manner. We introduce
the Google Research Football Environment, a new reinforcement learning
environment where agents are trained to play football in an advanced,
physics-based 3D simulator. The resulting environment is challenging, easy to
use and customize, and it is available under a permissive open-source license.
In addition, it provides support for multiplayer and multi-agent experiments.
We propose three full-game scenarios of varying difficulty with the Football
Benchmarks and report baseline results for three commonly used reinforcement
algorithms (IMPALA, PPO, and Ape-X DQN). We also provide a diverse set of
simpler scenarios with the Football Academy and showcase several promising
research directions.
