  Online continual learning, the process of training models on streaming data,
has gained increasing attention in recent years. However, a critical aspect
often overlooked is the label delay, where new data may not be labeled due to
slow and costly annotation processes. We introduce a new continual learning
framework with explicit modeling of the label delay between data and label
streams over time steps. In each step, the framework reveals both unlabeled
data from the current time step $t$ and labels delayed with $d$ steps, from the
time step $t-d$. In our extensive experiments amounting to 1060 GPU days, we
show that merely augmenting the computational resources is insufficient to
tackle this challenge. Our findings underline a notable performance decline
when solely relying on labeled data when the label delay becomes significant.
More surprisingly, when using state-of-the-art SSL and TTA techniques to
utilize the newer, unlabeled data, they fail to surpass the performance of a
na\"ive method that simply trains on the delayed supervised stream. To this
end, we introduce a simple, efficient baseline that rehearses from the labeled
memory samples that are most similar to the new unlabeled samples. This method
bridges the accuracy gap caused by label delay without significantly increasing
computational complexity. We show experimentally that our method is the least
affected by the label delay factor and in some cases successfully recovers the
accuracy of the non-delayed counterpart. We conduct various ablations and
sensitivity experiments, demonstrating the effectiveness of our approach.
