  For models consisting of a classifier in some representation space, learning
online from a non-stationary data stream often necessitates changes in the
representation. So, the question arises of what is the best way to adapt the
classifier to shifts in representation. Current methods only slowly change the
classifier to representation shift, introducing noise into learning as the
classifier is misaligned to the representation. We propose DeepCCG, an
empirical Bayesian approach to solve this problem. DeepCCG works by updating
the posterior of a class conditional Gaussian classifier such that the
classifier adapts in one step to representation shift. The use of a class
conditional Gaussian classifier also enables DeepCCG to use a log conditional
marginal likelihood loss to update the representation. To perform the update to
the classifier and representation, DeepCCG maintains a fixed number of examples
in memory and so a key part of DeepCCG is selecting what examples to store,
choosing the subset that minimises the KL divergence between the true posterior
and the posterior induced by the subset. We explore the behaviour of DeepCCG in
online continual learning (CL), demonstrating that it performs well against a
spectrum of online CL methods and that it reduces the change in performance due
to representation shift.
