  Reinforcement learning is an active research area with a vast number of
applications in robotics, and the RoboCup competition is an interesting
environment for studying and evaluating reinforcement learning methods. A known
difficulty in applying reinforcement learning to robotics is the high number of
experience samples required, being the use of simulated environments for
training the agents followed by transfer learning to real-world (sim-to-real) a
viable path. This article introduces an open-source simulator for the IEEE Very
Small Size Soccer and the Small Size League optimized for reinforcement
learning experiments. We also propose a framework for creating OpenAI Gym
environments with a set of benchmarks tasks for evaluating single-agent and
multi-agent robot soccer skills. We then demonstrate the learning capabilities
of two state-of-the-art reinforcement learning methods as well as their
limitations in certain scenarios introduced in this framework. We believe this
will make it easier for more teams to compete in these categories using
end-to-end reinforcement learning approaches and further develop this research
area.
