  Intrinsic capability to continuously learn a changing data stream is a
desideratum of deep neural networks (DNNs). However, current DNNs suffer from
catastrophic forgetting, which hinders remembering past knowledge. To mitigate
this issue, existing Continual Learning (CL) approaches either retain exemplars
for replay, regularize learning, or allocate dedicated capacity for new tasks.
This paper investigates an unexplored CL direction for incremental learning
called Incremental Latent Rectification or ILR. In a nutshell, ILR learns to
propagate with correction (or rectify) the representation from the current
trained DNN backward to the representation space of the old task, where
performing predictive decisions is easier. This rectification process only
employs a chain of small representation mapping networks, called rectifier
units. Empirical experiments on several continual learning benchmarks,
including CIFAR10, CIFAR100, and Tiny ImageNet, demonstrate the effectiveness
and potential of this novel CL direction compared to existing representative CL
methods.
