  Neural networks often suffer from catastrophic interference (CI): performance
on previously learned tasks drops off significantly when learning a new task.
This contrasts strongly with humans, who can sequentially learn new tasks
without appreciably forgetting previous tasks. Prior work has explored various
techniques for mitigating CI such as regularization, rehearsal, generative
replay, and distillation methods. The current work takes a different approach,
one guided by cognitive science research showing that in naturalistic
environments, the probability of encountering a task decreases as a power-law
of the time since it was last performed. We argue that a realistic evaluation
of techniques for the mitigation of CI should be performed in simulated
naturalistic learning environments. Thus, we evaluate the extent of mitigation
of CI when training simple rehearsal-based methods in power-law environments
similar to the ones humans face. Our work explores this novel rehearsal-based
approach for a domain-incremental task: learning permutations in the MNIST
task. We compare our rehearsal environment with other baselines to show its
efficacy in promoting continual learning. Additionally, we investigate whether
this environment shows forward facilitation, i.e., faster learning of later
tasks. Next, we explore the robustness of our learning environment to the
number of tasks, model size, and amount of data rehearsed after each task.
Notably, our results show that the performance is comparable or superior to
that of models trained using popular regularization methods and also to
rehearsals in non-power-law environments. The benefits of this training
paradigm include simplicity and the lack of a need for extra neural circuitry.
In addition, because our method is orthogonal to other methods, future research
can combine training in power-law environments with other continual learning
mechanisms.
