  The effectiveness of large language models (LLMs) is often hindered by
duplicated data in their extensive pre-training datasets. Current approaches
primarily focus on detecting and removing duplicates, which risks the loss of
valuable information and neglects the varying degrees of duplication. To
address this, we propose a soft deduplication method that maintains dataset
integrity while selectively reducing the sampling weight of data with high
commonness. Central to our approach is the concept of "data commonness", a
metric we introduce to quantify the degree of duplication by measuring the
occurrence probabilities of samples using an n-gram model. Empirical analysis
shows that this method significantly improves training efficiency, achieving
comparable perplexity scores with at least a 26% reduction in required training
steps. Additionally, it enhances average few-shot downstream accuracy by 1.77%
when trained for an equivalent duration. Importantly, this approach
consistently improves performance, even on rigorously deduplicated datasets,
indicating its potential to complement existing methods and become a standard
pre-training process for LLMs.
