  Language is constantly changing and evolving, leaving language models to
become quickly outdated. Consequently, we should continuously update our models
with new data to expose them to new events and facts. However, that requires
additional computing, which means new carbon emissions. Do any measurable
benefits justify this cost? This paper looks for empirical evidence to support
continuous training. We reproduce existing benchmarks and extend them to
include additional time periods, models, and tasks. Our results show that the
downstream task performance of temporally adapted English models for social
media data do not improve over time. Pretrained models without temporal
adaptation are actually significantly more effective and efficient. However, we
also note a lack of suitable temporal benchmarks. Our findings invite a
critical reflection on when and how to temporally adapt language models,
accounting for sustainability.
