  Real world learning scenarios involve a nonstationary distribution of classes
with sequential dependencies among the samples, in contrast to the standard
machine learning formulation of drawing samples independently from a fixed,
typically uniform distribution. Furthermore, real world interactions demand
learning on-the-fly from few or no class labels. In this work, we propose an
unsupervised model that simultaneously performs online visual representation
learning and few-shot learning of new categories without relying on any class
labels. Our model is a prototype-based memory network with a control component
that determines when to form a new class prototype. We formulate it as an
online mixture model, where components are created with only a single new
example, and assignments do not have to be balanced, which permits an
approximation to natural imbalanced distributions from uncurated raw data.
Learning includes a contrastive loss that encourages different views of the
same image to be assigned to the same prototype. The result is a mechanism that
forms categorical representations of objects in nonstationary environments.
Experiments show that our method can learn from an online stream of visual
input data and its learned representations are significantly better at category
recognition compared to state-of-the-art self-supervised learning methods.
