  Transformers have become the dominant architecture for sequence modeling
tasks such as natural language processing or audio processing, and they are now
even considered for tasks that are not naturally sequential such as image
classification. Their ability to attend to and to process a set of tokens as
context enables them to develop in-context few-shot learning abilities.
However, their potential for online continual learning remains relatively
unexplored. In online continual learning, a model must adapt to a
non-stationary stream of data, minimizing the cumulative nextstep prediction
loss. We focus on the supervised online continual learning setting, where we
learn a predictor $x_t \rightarrow y_t$ for a sequence of examples $(x_t,
y_t)$. Inspired by the in-context learning capabilities of transformers and
their connection to meta-learning, we propose a method that leverages these
strengths for online continual learning. Our approach explicitly conditions a
transformer on recent observations, while at the same time online training it
with stochastic gradient descent, following the procedure introduced with
Transformer-XL. We incorporate replay to maintain the benefits of multi-epoch
training while adhering to the sequential protocol. We hypothesize that this
combination enables fast adaptation through in-context learning and sustained
longterm improvement via parametric learning. Our method demonstrates
significant improvements over previous state-of-the-art results on CLOC, a
challenging large-scale real-world benchmark for image geo-localization.
