  We present a novel class incremental learning approach based on deep neural
networks, which continually learns new tasks with limited memory for storing
examples in the previous tasks. Our algorithm is based on knowledge
distillation and provides a principled way to maintain the representations of
old models while adjusting to new tasks effectively. The proposed method
estimates the relationship between the representation changes and the resulting
loss increases incurred by model updates. It minimizes the upper bound of the
loss increases using the representations, which exploits the estimated
importance of each feature map within a backbone model. Based on the
importance, the model restricts updates of important features for robustness
while allowing changes in less critical features for flexibility. This
optimization strategy effectively alleviates the notorious catastrophic
forgetting problem despite the limited accessibility of data in the previous
tasks. The experimental results show significant accuracy improvement of the
proposed algorithm over the existing methods on the standard datasets. Code is
available.
