  This article introduces an open framework, called VSSS-RL, for studying
Reinforcement Learning (RL) and sim-to-real in robot soccer, focusing on the
IEEE Very Small Size Soccer (VSSS) league. We propose a simulated environment
in which continuous or discrete control policies can be trained to control the
complete behavior of soccer agents and a sim-to-real method based on domain
adaptation to adapt the obtained policies to real robots. Our results show that
the trained policies learned a broad repertoire of behaviors that are difficult
to implement with handcrafted control policies. With VSSS-RL, we were able to
beat human-designed policies in the 2019 Latin American Robotics Competition
(LARC), achieving 4th place out of 21 teams, being the first to apply
Reinforcement Learning (RL) successfully in this competition. Both environment
and hardware specifications are available open-source to allow reproducibility
of our results and further studies.
