  Current deep learning classifiers, carry out supervised learning and store
class discriminatory information in a set of shared network weights. These
weights cannot be easily altered to incrementally learn additional classes,
since the classification weights all require retraining to prevent old class
information from being lost and also require the previous training data to be
present. We present a novel two stage architecture which couples visual feature
learning with probabilistic models to represent each class in the form of a
Gaussian Mixture Model. By using these independent class representations within
our classifier, we outperform a benchmark of an equivalent network with a
Softmax head, obtaining increased accuracy for sample sizes smaller than 12 and
increased weighted F1 score for 3 imbalanced class profiles in that sample
range. When learning new classes our classifier exhibits no catastrophic
forgetting issues and only requires the new classes' training images to be
present. This enables a database of growing classes over time which can be
visually indexed and reasoned over.
