  Experience replay is ubiquitous in reinforcement learning, to reuse past data
and improve sample efficiency. Though a variety of smart sampling schemes have
been introduced to improve performance, uniform sampling by far remains the
most common approach. One exception is Prioritized Experience Replay (PER),
where sampling is done proportionally to TD errors, inspired by the success of
prioritized sweeping in dynamic programming. The original work on PER showed
improvements in Atari, but follow-up results are mixed. In this paper, we
investigate several variations on PER, to attempt to understand where and when
PER may be useful. Our findings in prediction tasks reveal that while PER can
improve value propagation in tabular settings, behavior is significantly
different when combined with neural networks. Certain mitigations -- like
delaying target network updates to control generalization and using estimates
of expected TD errors in PER to avoid chasing stochasticity -- can avoid large
spikes in error with PER and neural networks, but nonetheless generally do not
outperform uniform replay. In control tasks, none of the prioritized variants
consistently outperform uniform replay.
