  Continually learning new classes from fresh data without forgetting previous
knowledge of old classes is a very challenging research problem. Moreover, it
is imperative that such learning must respect certain memory and computational
constraints such as (i) training samples are limited to only a few per class,
(ii) the computational cost of learning a novel class remains constant, and
(iii) the memory footprint of the model grows at most linearly with the number
of classes observed. To meet the above constraints, we propose C-FSCIL, which
is architecturally composed of a frozen meta-learned feature extractor, a
trainable fixed-size fully connected layer, and a rewritable dynamically
growing memory that stores as many vectors as the number of encountered
classes. C-FSCIL provides three update modes that offer a trade-off between
accuracy and compute-memory cost of learning novel classes. C-FSCIL exploits
hyperdimensional embedding that allows to continually express many more classes
than the fixed dimensions in the vector space, with minimal interference. The
quality of class vector representations is further improved by aligning them
quasi-orthogonally to each other by means of novel loss functions. Experiments
on the CIFAR100, miniImageNet, and Omniglot datasets show that C-FSCIL
outperforms the baselines with remarkable accuracy and compression. It also
scales up to the largest problem size ever tried in this few-shot setting by
learning 423 novel classes on top of 1200 base classes with less than 1.6%
accuracy drop. Our code is available at
https://github.com/IBM/constrained-FSCIL.
