  Continual learning is the problem of learning new tasks or knowledge while
protecting old knowledge and ideally generalizing from old experience to learn
new tasks faster. Neural networks trained by stochastic gradient descent often
degrade on old tasks when trained successively on new tasks with different data
distributions. This phenomenon, referred to as catastrophic forgetting, is
considered a major hurdle to learning with non-stationary data or sequences of
new tasks, and prevents networks from continually accumulating knowledge and
skills. We examine this issue in the context of reinforcement learning, in a
setting where an agent is exposed to tasks in a sequence. Unlike most other
work, we do not provide an explicit indication to the model of task boundaries,
which is the most general circumstance for a learning agent exposed to
continuous experience. While various methods to counteract catastrophic
forgetting have recently been proposed, we explore a straightforward, general,
and seemingly overlooked solution - that of using experience replay buffers for
all past events - with a mixture of on- and off-policy learning, leveraging
behavioral cloning. We show that this strategy can still learn new tasks
quickly yet can substantially reduce catastrophic forgetting in both Atari and
DMLab domains, even matching the performance of methods that require task
identities. When buffer storage is constrained, we confirm that a simple
mechanism for randomly discarding data allows a limited size buffer to perform
almost as well as an unbounded one.
