  It is a widely accepted principle that software without tests has bugs.
Testing reinforcement learning agents is especially difficult because of the
stochastic nature of both agents and environments, the complexity of
state-of-the-art models, and the sequential nature of their predictions.
Recently, the Arcade Learning Environment (ALE) has become one of the most
widely used benchmark suites for deep learning research, and state-of-the-art
Reinforcement Learning (RL) agents have been shown to routinely equal or exceed
human performance on many ALE tasks. Since ALE is based on emulation of
original Atari games, the environment does not provide semantically meaningful
representations of internal game state. This means that ALE has limited utility
as an environment for supporting testing or model introspection. We propose
ToyBox, a collection of reimplementations of these games that solves this
critical problem and enables robust testing of RL agents.
