  The ability to dynamically adapt neural networks to newly-available data
without performance deterioration would revolutionize deep learning
applications. Streaming learning (i.e., learning from one data example at a
time) has the potential to enable such real-time adaptation, but current
approaches i) freeze a majority of network parameters during streaming and ii)
are dependent upon offline, base initialization procedures over large subsets
of data, which damages performance and limits applicability. To mitigate these
shortcomings, we propose Cold Start Streaming Learning (CSSL), a simple,
end-to-end approach for streaming learning with deep networks that uses a
combination of replay and data augmentation to avoid catastrophic forgetting.
  Because CSSL updates all model parameters during streaming, the algorithm is
capable of beginning streaming from a random initialization, making base
initialization optional. Going further, the algorithm's simplicity allows
theoretical convergence guarantees to be derived using analysis of the Neural
Tangent Random Feature (NTRF). In experiments, we find that CSSL outperforms
existing baselines for streaming learning in experiments on CIFAR100, ImageNet,
and Core50 datasets. Additionally, we propose a novel multi-task streaming
learning setting and show that CSSL performs favorably in this domain. Put
simply, CSSL performs well and demonstrates that the complicated, multi-step
training pipelines adopted by most streaming methodologies can be replaced with
a simple, end-to-end learning approach without sacrificing performance.
