  Increasing the scale of reinforcement learning experiments has allowed
researchers to achieve unprecedented results in both training sophisticated
agents for video games, and in sim-to-real transfer for robotics. Typically
such experiments rely on large distributed systems and require expensive
hardware setups, limiting wider access to this exciting area of research. In
this work we aim to solve this problem by optimizing the efficiency and
resource utilization of reinforcement learning algorithms instead of relying on
distributed computation. We present the "Sample Factory", a high-throughput
training system optimized for a single-machine setting. Our architecture
combines a highly efficient, asynchronous, GPU-based sampler with off-policy
correction techniques, allowing us to achieve throughput higher than $10^5$
environment frames/second on non-trivial control problems in 3D without
sacrificing sample efficiency. We extend Sample Factory to support self-play
and population-based training and apply these techniques to train highly
capable agents for a multiplayer first-person shooter game. The source code is
available at https://github.com/alex-petrenko/sample-factory
