  Language Models (LMs) pre-trained with self-supervision on large text corpora
have become the default starting point for developing models for various NLP
tasks. Once the pre-training corpus has been assembled, all data samples in the
corpus are treated with equal importance during LM pre-training. However, due
to varying levels of relevance and quality of data, equal importance to all the
data samples may not be the optimal choice. While data reweighting has been
explored in the context of task-specific supervised learning and LM
fine-tuning, model-driven reweighting for pre-training data has not been
explored. We fill this important gap and propose PRESENCE, a method for jointly
reweighting samples by leveraging self-influence (SI) scores as an indicator of
sample importance and pre-training. PRESENCE promotes novelty and stability for
model pre-training. Through extensive analysis spanning multiple model sizes,
datasets, and tasks, we present PRESENCE as an important first step in the
research direction of sample reweighting for pre-training language models.
