  Prioritized experience replay is a reinforcement learning technique whereby
agents speed up learning by replaying useful past experiences. This usefulness
is quantified as the expected gain from replaying the experience, a quantity
often approximated as the prediction error (TD-error). However, recent work in
neuroscience suggests that, in biological organisms, replay is prioritized not
only by gain, but also by "need" -- a quantity measuring the expected relevance
of each experience with respect to the current situation. Importantly, this
term is not currently considered in algorithms such as prioritized experience
replay. In this paper we present a new approach for prioritizing experiences
for replay that considers both gain and need. Our proposed algorithms show a
significant increase in performance in benchmarks including the Dyna-Q maze and
a selection of Atari games.
