  Experience replay \citep{lin1993reinforcement, mnih2015human} is a widely
used technique to achieve efficient use of data and improved performance in RL
algorithms. In experience replay, past transitions are stored in a memory
buffer and re-used during learning. Various suggestions for sampling schemes
from the replay buffer have been suggested in previous works, attempting to
optimally choose those experiences which will most contribute to the
convergence to an optimal policy. Here, we give some conditions on the replay
sampling scheme that will ensure convergence, focusing on the well-known
Q-learning algorithm in the tabular setting. After establishing sufficient
conditions for convergence, we turn to suggest a slightly different usage for
experience replay - replaying memories in a biased manner as a means to change
the properties of the resulting policy. We initiate a rigorous study of
experience replay as a tool to control and modify the properties of the
resulting policy. In particular, we show that using an appropriate biased
sampling scheme can allow us to achieve a \emph{safe} policy. We believe that
using experience replay as a biasing mechanism that allows controlling the
resulting policy in desirable ways is an idea with promising potential for many
applications.
