  Continual learning (CL) aims to continually accumulate knowledge from a
non-stationary data stream without catastrophic forgetting of learned
knowledge, requiring a balance between stability and adaptability. Relying on
the generalizable representation in pre-trained models (PTMs), PTM-based CL
methods perform effective continual adaptation on downstream tasks by adding
learnable adapters or prompts upon the frozen PTMs. However, many existing
PTM-based CL methods use restricted adaptation on a fixed set of these modules
to avoid forgetting, suffering from limited CL ability. Periodically adding
task-specific modules results in linear model growth rate and impaired
knowledge reuse. We propose Self-Expansion of pre-trained models with
Modularized Adaptation (SEMA), a novel approach to enhance the control of
stability-plasticity balance in PTM-based CL. SEMA automatically decides to
reuse or add adapter modules on demand in CL, depending on whether significant
distribution shift that cannot be handled is detected at different
representation levels. We design modular adapter consisting of a functional
adapter and a representation descriptor. The representation descriptors are
trained as a distribution shift indicator and used to trigger self-expansion
signals. For better composing the adapters, an expandable weighting router is
learned jointly for mixture of adapter outputs. SEMA enables better knowledge
reuse and sub-linear expansion rate. Extensive experiments demonstrate the
effectiveness of the proposed self-expansion method, achieving state-of-the-art
performance compared to PTM-based CL methods without memory rehearsal.
