  A plain well-trained deep learning model often does not have the ability to
learn new knowledge without forgetting the previously learned knowledge, which
is known as catastrophic forgetting. Here we propose a novel method,
SupportNet, to efficiently and effectively solve the catastrophic forgetting
problem in the class incremental learning scenario. SupportNet combines the
strength of deep learning and support vector machine (SVM), where SVM is used
to identify the support data from the old data, which are fed to the deep
learning model together with the new data for further training so that the
model can review the essential information of the old data when learning the
new information. Two powerful consolidation regularizers are applied to
stabilize the learned representation and ensure the robustness of the learned
model. We validate our method with comprehensive experiments on various tasks,
which show that SupportNet drastically outperforms the state-of-the-art
incremental learning methods and even reaches similar performance as the deep
learning model trained from scratch on both old and new data. Our program is
accessible at: https://github.com/lykaust15/SupportNet
