  In real-world environments, continual learning is essential for machine
learning models, as they need to acquire new knowledge incrementally without
forgetting what they have already learned. While pretrained language models
have shown impressive capabilities on various static tasks, applying them to
continual learning poses significant challenges, including avoiding
catastrophic forgetting, facilitating knowledge transfer, and maintaining
parameter efficiency. In this paper, we introduce MoCL-P, a novel lightweight
continual learning method that addresses these challenges simultaneously.
Unlike traditional approaches that continuously expand parameters for newly
arriving tasks, MoCL-P integrates task representation-guided module composition
with adaptive pruning, effectively balancing knowledge integration and
computational overhead. Our evaluation across three continual learning
benchmarks with up to 176 tasks shows that MoCL-P achieves state-of-the-art
performance and improves parameter efficiency by up to three times,
demonstrating its potential for practical applications where resource
requirements are constrained.
