  Continual learning aims at incrementally acquiring new knowledge while not
forgetting existing knowledge. To overcome catastrophic forgetting, methods are
either rehearsal-based, i.e., store data examples from previous tasks for data
replay, or isolate parameters dedicated to each task. However, rehearsal-based
methods raise privacy and memory issues, and parameter-isolation continual
learning does not consider interaction between tasks, thus hindering knowledge
transfer. In this work, we propose MoCL, a rehearsal-free Modular and
Compositional Continual Learning framework which continually adds new modules
to language models and composes them with existing modules. Experiments on
various benchmarks show that MoCL outperforms state of the art and effectively
facilitates knowledge transfer.
