  When an agent acquires new information, ideally it would immediately be
capable of using that information to understand its environment. This is not
possible using conventional deep neural networks, which suffer from
catastrophic forgetting when they are incrementally updated, with new knowledge
overwriting established representations. A variety of approaches have been
developed that attempt to mitigate catastrophic forgetting in the incremental
batch learning scenario, where a model learns from a series of large
collections of labeled samples. However, in this setting, inference is only
possible after a batch has been accumulated, which prohibits many applications.
An alternative paradigm is online learning in a single pass through the
training dataset on a resource constrained budget, which is known as streaming
learning. Streaming learning has been much less studied in the deep learning
community. In streaming learning, an agent learns instances one-by-one and can
be tested at any time, rather than only after learning a large batch. Here, we
revisit streaming linear discriminant analysis, which has been widely used in
the data mining research community. By combining streaming linear discriminant
analysis with deep learning, we are able to outperform both incremental batch
learning and streaming learning algorithms on both ImageNet ILSVRC-2012 and
CORe50, a dataset that involves learning to classify from temporally ordered
samples.
