  Experience replay is a key technique behind many recent advances in deep
reinforcement learning. Allowing the agent to learn from earlier memories can
speed up learning and break undesirable temporal correlations. Despite its
wide-spread application, very little is understood about the properties of
experience replay. How does the amount of memory kept affect learning dynamics?
Does it help to prioritize certain experiences? In this paper, we address these
questions by formulating a dynamical systems ODE model of Q-learning with
experience replay. We derive analytic solutions of the ODE for a simple
setting. We show that even in this very simple setting, the amount of memory
kept can substantially affect the agent's performance. Too much or too little
memory both slow down learning. Moreover, we characterize regimes where
prioritized replay harms the agent's learning. We show that our analytic
solutions have excellent agreement with experiments. Finally, we propose a
simple algorithm for adaptively changing the memory buffer size which achieves
consistently good empirical performance.
