  While many multi-robot coordination problems can be solved optimally by exact
algorithms, solutions are often not scalable in the number of robots.
Multi-Agent Reinforcement Learning (MARL) is gaining increasing attention in
the robotics community as a promising solution to tackle such problems.
Nevertheless, we still lack the tools that allow us to quickly and efficiently
find solutions to large-scale collective learning tasks. In this work, we
introduce the Vectorized Multi-Agent Simulator (VMAS). VMAS is an open-source
framework designed for efficient MARL benchmarking. It is comprised of a
vectorized 2D physics engine written in PyTorch and a set of twelve challenging
multi-robot scenarios. Additional scenarios can be implemented through a simple
and modular interface. We demonstrate how vectorization enables parallel
simulation on accelerated hardware without added complexity. When comparing
VMAS to OpenAI MPE, we show how MPE's execution time increases linearly in the
number of simulations while VMAS is able to execute 30,000 parallel simulations
in under 10s, proving more than 100x faster. Using VMAS's RLlib interface, we
benchmark our multi-robot scenarios using various Proximal Policy Optimization
(PPO)-based MARL algorithms. VMAS's scenarios prove challenging in orthogonal
ways for state-of-the-art MARL algorithms. The VMAS framework is available at
https://github.com/proroklab/VectorizedMultiAgentSimulator. A video of VMAS
scenarios and experiments is available at https://youtu.be/aaDRYfiesAY.
