  Deep Q-Network (DQN) marked a major milestone for reinforcement learning,
demonstrating for the first time that human-level control policies could be
learned directly from raw visual inputs via reward maximization. Even years
after its introduction, DQN remains highly relevant to the research community
since many of its innovations have been adopted by successor methods.
Nevertheless, despite significant hardware advances in the interim, DQN's
original Atari 2600 experiments remain costly to replicate in full. This poses
an immense barrier to researchers who cannot afford state-of-the-art hardware
or lack access to large-scale cloud computing resources. To facilitate improved
access to deep reinforcement learning research, we introduce a DQN
implementation that leverages a novel concurrent and synchronized execution
framework designed to maximally utilize a heterogeneous CPU-GPU desktop system.
With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the
training time of a 200-million-frame Atari experiment from 25 hours to just 9
hours. The ideas introduced in our paper should be generalizable to a large
number of off-policy deep reinforcement learning methods.
