  Lifelong learning or continual learning is the problem of training an AI
agent continuously while also preventing it from forgetting its previously
acquired knowledge. Streaming lifelong learning is a challenging setting of
lifelong learning with the goal of continuous learning in a dynamic
non-stationary environment without forgetting. We introduce a novel approach to
lifelong learning, which is streaming (observes each training example only
once), requires a single pass over the data, can learn in a class-incremental
manner, and can be evaluated on-the-fly (anytime inference). To accomplish
these, we propose a novel \emph{virtual gradients} based approach for continual
representation learning which adapts to each new example while also
generalizing well on past data to prevent catastrophic forgetting. Our approach
also leverages an exponential-moving-average-based semantic memory to further
enhance performance. Experiments on diverse datasets with temporally correlated
observations demonstrate our method's efficacy and superior performance over
existing methods.
