  Recent research has revealed that neural language models at scale suffer from
poor temporal generalization capability, i.e., the language model pre-trained
on static data from past years performs worse over time on emerging data.
Existing methods mainly perform continual training to mitigate such a
misalignment. While effective to some extent but is far from being addressed on
both the language modeling and downstream tasks. In this paper, we empirically
observe that temporal generalization is closely affiliated with lexical
semantic change, which is one of the essential phenomena of natural languages.
Based on this observation, we propose a simple yet effective lexical-level
masking strategy to post-train a converged language model. Experiments on two
pre-trained language models, two different classification tasks, and four
benchmark datasets demonstrate the effectiveness of our proposed method over
existing temporal adaptation methods, i.e., continual training with new data.
Our code is available at \url{https://github.com/zhaochen0110/LMLM}.
