  Deep neural networks often severely forget previously learned knowledge when
learning new knowledge. Various continual learning (CL) methods have been
proposed to handle such a catastrophic forgetting issue from different
perspectives and achieved substantial improvements. In this paper, a novel
two-branch continual learning framework is proposed to further enhance most
existing CL methods. Specifically, the main branch can be any existing CL model
and the newly introduced side branch is a lightweight convolutional network.
The output of each main branch block is modulated by the output of the
corresponding side branch block. Such a simple two-branch model can then be
easily implemented and learned with the vanilla optimization setting without
whistles and bells. Extensive experiments with various settings on multiple
image datasets show that the proposed framework yields consistent improvements
over state-of-the-art methods.
