  Replay methods are known to be successful at mitigating catastrophic
forgetting in continual learning scenarios despite having limited access to
historical data. However, storing historical data is cheap in many real-world
settings, yet replaying all historical data is often prohibited due to
processing time constraints. In such settings, we propose that continual
learning systems should learn the time to learn and schedule which tasks to
replay at different time steps. We first demonstrate the benefits of our
proposal by using Monte Carlo tree search to find a proper replay schedule, and
show that the found replay schedules can outperform fixed scheduling policies
when combined with various replay methods in different continual learning
settings. Additionally, we propose a framework for learning replay scheduling
policies with reinforcement learning. We show that the learned policies can
generalize better in new continual learning scenarios compared to equally
replaying all seen tasks, without added computational cost. Our study reveals
the importance of learning the time to learn in continual learning, which
brings current research closer to real-world needs.
