  This paper aims to understand the impacts of various data combinations (e.g.,
web text, Wikipedia, GitHub, books) on the pretraining of large language models
using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
dataset, which has been refined and further deduplicated to 627B tokens from
the extensive 1.2T token RedPajama dataset contributed by Together. We have
termed our research as SlimPajama-DC, an empirical analysis designed to uncover
fundamental characteristics and best practices associated with employing
SlimPajama in the training of large language models. During our research with
SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
local deduplication. We analyze and discuss how global (across different
sources of datasets) and local (within the single source of dataset)
deduplications affect the performance of trained models. (2) Proportions of
highly-deduplicated multi-source datasets in the combination. To study this, we
construct six configurations on SlimPajama dataset and train individual ones
using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best configuration
outperforms the 1.3B model trained on RedPajama using the same number of
training tokens by a significant margin. All our 1.3B models are trained on
Cerebras 16$\times$ CS-2 cluster with a total of 80 PFLOP/s in bf16 mixed
precision. We further extend our discoveries (such as increasing data diversity
is crucial after global deduplication) on a 7B model with large batch-size
training. Our SlimPajama-DC models are available at:
https://huggingface.co/MBZUAI-LLM/SlimPajama-DC and the separate SlimPajama-DC
datasets are available at:
https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC.
