  This paper introduces a novel method for learning how to play the most
difficult Atari 2600 games from the Arcade Learning Environment using deep
reinforcement learning. The proposed method, human checkpoint replay, consists
in using checkpoints sampled from human gameplay as starting points for the
learning process. This is meant to compensate for the difficulties of current
exploration strategies, such as epsilon-greedy, to find successful control
policies in games with sparse rewards. Like other deep reinforcement learning
architectures, our model uses a convolutional neural network that receives only
raw pixel inputs to estimate the state value function. We tested our method on
Montezuma's Revenge and Private Eye, two of the most challenging games from the
Atari platform. The results we obtained show a substantial improvement compared
to previous learning approaches, as well as over a random player. We also
propose a method for training deep reinforcement learning agents using human
gameplay experience, which we call human experience replay.
