  We introduce Procgen Benchmark, a suite of 16 procedurally generated
game-like environments designed to benchmark both sample efficiency and
generalization in reinforcement learning. We believe that the community will
benefit from increased access to high quality training environments, and we
provide detailed experimental protocols for using this benchmark. We
empirically demonstrate that diverse environment distributions are essential to
adequately train and evaluate RL agents, thereby motivating the extensive use
of procedural content generation. We then use this benchmark to investigate the
effects of scaling model size, finding that larger models significantly improve
both sample efficiency and generalization.
