  Experience replay (ER) is a fundamental component of off-policy deep
reinforcement learning (RL). ER recalls experiences from past iterations to
compute gradient estimates for the current policy, increasing data-efficiency.
However, the accuracy of such updates may deteriorate when the policy diverges
from past behaviors and can undermine the performance of ER. Many algorithms
mitigate this issue by tuning hyper-parameters to slow down policy changes. An
alternative is to actively enforce the similarity between policy and the
experiences in the replay memory. We introduce Remember and Forget Experience
Replay (ReF-ER), a novel method that can enhance RL algorithms with
parameterized policies. ReF-ER (1) skips gradients computed from experiences
that are too unlikely with the current policy and (2) regulates policy changes
within a trust region of the replayed behaviors. We couple ReF-ER with
Q-learning, deterministic policy gradient and off-policy gradient methods. We
find that ReF-ER consistently improves the performance of continuous-action,
off-policy RL on fully observable benchmarks and partially observable flow
control problems.
