  Real-world applications often require learning continuously from a stream of
data under ever-changing conditions. When trying to learn from such
non-stationary data, deep neural networks (DNNs) undergo catastrophic
forgetting of previously learned information. Among the common approaches to
avoid catastrophic forgetting, rehearsal-based methods have proven effective.
However, they are still prone to forgetting due to task-interference as all
parameters respond to all tasks. To counter this, we take inspiration from
sparse coding in the brain and introduce dynamic modularity and sparsity
(Dynamos) for rehearsal-based general continual learning. In this setup, the
DNN learns to respond to stimuli by activating relevant subsets of neurons. We
demonstrate the effectiveness of Dynamos on multiple datasets under challenging
continual learning evaluation protocols. Finally, we show that our method
learns representations that are modular and specialized, while maintaining
reusability by activating subsets of neurons with overlaps corresponding to the
similarity of stimuli.
