  In machine learning, temporal shifts occur when there are differences between
training and test splits in terms of time. For streaming data such as news or
social media, models are commonly trained on a fixed corpus from a certain
period of time, and they can become obsolete due to the dynamism and evolving
nature of online content. This paper focuses on temporal shifts in social media
and, in particular, Twitter. We propose a unified evaluation scheme to assess
the performance of language models (LMs) under temporal shift on standard
social media tasks. LMs are tested on five diverse social media NLP tasks under
different temporal settings, which revealed two important findings: (i) the
decrease in performance under temporal shift is consistent across different
models for entity-focused tasks such as named entity recognition or
disambiguation, and hate speech detection, but not significant in the other
tasks analysed (i.e., topic and sentiment classification); and (ii) continuous
pre-training on the test period does not improve the temporal adaptability of
LMs.
