  We introduce for continual learning Autodiff Quadratic Consolidation (AQC),
which approximates the previous loss function with a quadratic function, and
Neural Consolidation (NC), which approximates the previous loss function with a
neural network. Although they are not scalable to large neural networks, they
can be used with a fixed pre-trained feature extractor. We empirically study
these methods in class-incremental learning, for which regularization-based
methods produce unsatisfactory results, unless combined with replay. We find
that for small datasets, quadratic approximation of the previous loss function
leads to poor results, even with full Hessian computation, and NC could
significantly improve the predictive performance, while for large datasets,
when used with a fixed pre-trained feature extractor, AQC provides superior
predictive performance. We also find that using tanh-output features can
improve the predictive performance of AQC. In particular, in class-incremental
Split MNIST, when a Convolutional Neural Network (CNN) with tanh-output
features is pre-trained on EMNIST Letters and used as a fixed pre-trained
feature extractor, AQC can achieve predictive performance comparable to joint
training.
