  In the evolving field of Natural Language Processing, understanding the
temporal context of text is increasingly crucial. This study investigates
methods to incorporate temporal information during pre-training, aiming to
achieve effective time-aware language representation for improved performance
on time-related tasks. In contrast to common pre-trained models like BERT,
which rely on synchronic document collections such as BookCorpus and Wikipedia,
our research introduces BiTimeBERT 2.0, a novel language model pre-trained on a
temporal news article collection. BiTimeBERT 2.0 utilizes this temporal news
collection, focusing on three innovative pre-training objectives: Time-Aware
Masked Language Modeling (TAMLM), Document Dating (DD), and Time-Sensitive
Entity Replacement (TSER). Each objective targets a unique aspect of temporal
information. TAMLM is designed to enhance the understanding of temporal
contexts and relations, DD integrates document timestamps as chronological
markers, and TSER focuses on the temporal dynamics of "Person" entities,
recognizing their inherent temporal significance. The experimental results
consistently demonstrate that BiTimeBERT 2.0 outperforms models like BERT and
other existing pre-trained models, achieving substantial gains across a variety
of downstream NLP tasks and applications where time plays a pivotal role.
