  Continual RL requires an agent to learn new tasks without forgetting previous
ones, while improving on both past and future tasks. The most common approaches
use model-free algorithms and replay buffers can help to mitigate catastrophic
forgetting, but often struggle with scalability due to large memory
requirements. Biologically inspired replay suggests replay to a world model,
aligning with model-based RL; as opposed to the common setting of replay in
model-free algorithms. Model-based RL offers benefits for continual RL by
leveraging knowledge of the environment, independent of policy. We introduce
WMAR (World Models with Augmented Replay), a model-based RL algorithm with a
memory-efficient distribution-matching replay buffer. WMAR extends the well
known DreamerV3 algorithm, which employs a simple FIFO buffer and was not
tested in continual RL. We evaluated WMAR and DreamerV3, with the same-size
replay buffers. They were tested on two scenarios: tasks with shared structure
using OpenAI Procgen and tasks without shared structure using the Atari
benchmark. WMAR demonstrated favourable properties for continual RL considering
metrics for forgetting as well as skill transfer on past and future tasks.
Compared to DreamerV3, WMAR showed slight benefits in tasks with shared
structure and substantially better forgetting characteristics on tasks without
shared structure. Our results suggest that model-based RL with a
memory-efficient replay buffer can be an effective approach to continual RL,
justifying further research.
