  A shared goal of several machine learning communities like continual
learning, meta-learning and transfer learning, is to design algorithms and
models that efficiently and robustly adapt to unseen tasks. An even more
ambitious goal is to build models that never stop adapting, and that become
increasingly more efficient through time by suitably transferring the accrued
knowledge. Beyond the study of the actual learning algorithm and model
architecture, there are several hurdles towards our quest to build such models,
such as the choice of learning protocol, metric of success and data needed to
validate research hypotheses. In this work, we introduce the Never-Ending
VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of
over 100 visual classification tasks, sorted chronologically and extracted from
papers sampled uniformly from computer vision proceedings spanning the last
three decades. The resulting stream reflects what the research community
thought was meaningful at any point in time, and it serves as an ideal test bed
to assess how well models can adapt to new tasks, and do so better and more
efficiently as time goes by. Despite being limited to classification, the
resulting stream has a rich diversity of tasks from OCR, to texture analysis,
scene recognition, and so forth. The diversity is also reflected in the wide
range of dataset sizes, spanning over four orders of magnitude. Overall,
NEVIS'22 poses an unprecedented challenge for current sequential learning
approaches due to the scale and diversity of tasks, yet with a low entry
barrier as it is limited to a single modality and well understood supervised
learning problems. Moreover, we provide a reference implementation including
strong baselines and an evaluation protocol to compare methods in terms of
their trade-off between accuracy and compute.
