  Experience replay enables online reinforcement learning agents to store and
reuse the previous experiences of interacting with the environment. In the
original method, the experiences are sampled and replayed uniformly at random.
A prior work called prioritized experience replay was developed where
experiences are prioritized, so as to replay experiences seeming to be more
important more frequently. In this paper, we develop a method called
double-prioritized state-recycled (DPSR) experience replay, prioritizing the
experiences in both training stage and storing stage, as well as replacing the
experiences in the memory with state recycling to make the best of experiences
that seem to have low priorities temporarily. We used this method in Deep
Q-Networks (DQN), and achieved a state-of-the-art result, outperforming the
original method and prioritized experience replay on many Atari games.
