  The ability to learn new concepts continually is necessary in this
ever-changing world. However, deep neural networks suffer from catastrophic
forgetting when learning new categories. Many works have been proposed to
alleviate this phenomenon, whereas most of them either fall into the
stability-plasticity dilemma or take too much computation or storage overhead.
Inspired by the gradient boosting algorithm to gradually fit the residuals
between the target model and the previous ensemble model, we propose a novel
two-stage learning paradigm FOSTER, empowering the model to learn new
categories adaptively. Specifically, we first dynamically expand new modules to
fit the residuals between the target and the output of the original model.
Next, we remove redundant parameters and feature dimensions through an
effective distillation strategy to maintain the single backbone model. We
validate our method FOSTER on CIFAR-100 and ImageNet-100/1000 under different
settings. Experimental results show that our method achieves state-of-the-art
performance. Code is available at: https://github.com/G-U-N/ECCV22-FOSTER.
