  Continual learning is a learning paradigm that learns tasks sequentially with
resources constraints, in which the key challenge is stability-plasticity
dilemma, i.e., it is uneasy to simultaneously have the stability to prevent
catastrophic forgetting of old tasks and the plasticity to learn new tasks
well. In this paper, we propose a new continual learning approach, Advanced
Null Space (AdNS), to balance the stability and plasticity without storing any
old data of previous tasks. Specifically, to obtain better stability, AdNS
makes use of low-rank approximation to obtain a novel null space and projects
the gradient onto the null space to prevent the interference on the past tasks.
To control the generation of the null space, we introduce a non-uniform
constraint strength to further reduce forgetting. Furthermore, we present a
simple but effective method, intra-task distillation, to improve the
performance of the current task. Finally, we theoretically find that null space
plays a key role in plasticity and stability, respectively. Experimental
results show that the proposed method can achieve better performance compared
to state-of-the-art continual learning approaches.
