Announcing Nemotron-CC: A Trillion-Token English Language Dataset for LLM Pretraining
NVIDIA is excited to announce the release of Nemotron-CC, a 6.3-trillion-token English language
Common Crawl
dataset for pretraining highly accurate
large language models (LLMs)
, including 1.9 trillion tokens of synthetically generated data. One of the keys to training state-of-the-art LLMs is a high-quality pretraining dataset, and recent top LLMs, such as the
Meta Llama series,
were trained on vast amounts of data comprising 15 trillion tokens.
But little is known about the exact composition of these 15 trillion tokens. Nemotron-CC aims to remedy this and enable the wider community to train highly accurate LLMs. Internet crawl data, typically from Common Crawl, is generally the largest source of tokens. Recent open Common Crawl datasets, such as FineWeb-Edu and DCLM, have shown how to greatly improve benchmark accuracies over relatively short token horizons. However, this has been accomplished at the cost of removing 90% of data. This limits the suitability for long token horizon training, such as 15 trillion tokens for Llama 3.1.
Nemotron-CC fills this gap and shows how to transform Common Crawl data into a high-quality dataset suitable for training LLMs better than Llama 3.1 8B through a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters.
Results
Shown in Figure 1 are MMLU scores when training 8B parameter models for 1 trillion tokens, varying only the 73% English Common Crawl portion of the training data. Compared to the leading open English Common Crawl dataset DCLM, the high-quality subset Nemotron-CC-HQ increases the MMLU by +5.6.
Figure 1.
MMLU scores for 8B parameter models trained for 1 trillion tokens
Furthermore, the full 6.3-trillion-token dataset matches DCLM on MMLU, but contains four times more unique real tokens. This unlocks effective training over a long token horizon: an 8 billion parameter model trained for 15 trillion tokens, of which 7.2 trillion came from Nemotron-CC, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks.
Key insights
Some of the key insights that led to these results include:
Ensembling different model-based classifiers can help select a larger and more diverse set of high quality tokens.
Rephrasing can effectively reduce noise and errors in low-quality data and produce diverse variants with fresh unique tokens from high-quality data, leading to better results in downstream tasks.
Disabling traditional non-learned heuristic filters for high-quality data can further boost high quality token yield without hurting accuracy.
Data curation steps
Using
NVIDIA NeMo Curator
, we extracted and cleaned data from Common Crawl and then:
Filtered it for the English language
Performed global fuzzy deduplication as well as exact substring deduplication
Leveraged model-based filters such as DCLM, fineweb-edu for quality classification
Applied various heuristic and perplexity filters to further remove lower-quality data
We also leveraged synthetic data generation pipelines to generate ~2 trillion tokens of synthetic data.
The full recipe including the synthetic data generation pipelines will be merged into the
NVIDIA/NeMo-Curator
GitHub repo soon. To receive updates, star the repo.
Conclusion
Nemotron-CC
is an open, large, high-quality English Common Crawl dataset that enables pretraining highly accurate LLMs over both short and long token horizons. In the future, we hope to release more datasets that are key ingredients for state-of-the-art LLM pretraining, such as a specialized math pretraining dataset.
Download the dataset
from Common Crawl.
Use
NeMo Curator
to curate your own datasets.
Learn more about the technical details in
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset.
Acknowledgments
We thank the Common Crawl Foundation for hosting the dataset. We thank Pedro Ortiz Suarez for valuable feedback that improved the paper and Greg Lindahl for help with improving the data formatting and layout.
宣布推出 Nemotron-CC：用于 LLM 预训练的万亿级英语语言数据集
NVIDIA 很高兴地宣布发布 Nemotron-CC，这是一种包含 6.3 万亿个令牌的 英语语言
Common Crawl
数据集 ，用于预训练高度准确的
大语言模型 (LLMs)
，其中包括 1.9 万亿个令牌的合成生成数据。高质量预训练数据集是训练先进 LLM 的关键之一，近期的顶级 LLM (例如
Meta Llama 系列
) 基于包含 15 万亿个令牌的大量数据进行了训练。
但我们对这 15 万亿个令牌的确切构成知之甚少。Nemotron-CC 旨在解决这一问题，并使更广泛的社区能够训练高度准确的 LLM。互联网爬行数据 (通常来自 Common Crawl) 通常是最大的令牌来源。近期开放的 Common Crawl 数据集 (例如 FineWeb-Edu 和 DCLM) 展示了如何在相对较短的令牌范围内大幅提高基准精度。但是，实现此目标的代价是移除 90%的数据。这限制了长令牌水平训练的适用性，例如 Llama 3.1 的 15 万亿令牌。
Nemotron-CC 填补了这一空白，并展示了如何通过结合使用分类器集成、合成数据重述以及减少对启发式过滤器的依赖，将 Common Crawl 数据转换为比 Llama 3.1 8B 更好地训练 LLM 的高质量数据集。
成果
图 1 显示了为 1 万亿个令牌训练 8 亿个参数模型时的 MMLU 分数，该分数仅与训练数据的 73% 英文 Common Crawl 部分不同。与领先的开放式英文 Common Crawl 数据集 DCLM 相比，高质量子集 Nemotron-CC-HQ 可将 MMLU 增加约 5.6 倍。
图 1.
针对 1 万亿个令牌训练的 8 亿个参数模型的 MMLU 分数
此外，完整的 6.3 万亿 token 数据集与 MMLU 上的 DCLM 相匹配，但包含的唯一真实 token 是后者的四倍。这样一来，便可在漫长的令牌范围内实现有效训练：使用 15 万亿个令牌训练的 8 亿个参数模型 (其中 7.2 万亿个来自 Nemotron-CC) 优于 Llama 3.1 8B 模型：在 MMLU 上为 +5，在 ARC-Challenge 上为 +3.1，在 10 个不同的任务中平均为 +0.5。
主要见解
产生这些结果的一些关键见解包括：
集成基于模型的不同分类器有助于选择更大、更多样化的高质量 token 集。
重写可以有效减少低质量数据中的噪声和错误，并使用来自高质量数据的新鲜且独特的 token 生成各种变体，从而在下游任务中获得更好的结果。
禁用传统的非学习启发式过滤器可以在不影响准确性的情况下进一步提高高质量的 token 产量。
数据管护步骤
使用
NVIDIA NeMo Curator
，从 Common Crawl 中提取和清理数据，然后：
将其过滤为英语
执行全局模糊重复数据和精确的子字符串重复数据
利用基于模型的过滤器 (例如 DCLM、fineweb-edu) 进行质量分类
应用各种启发式和困惑性滤镜，进一步移除低质量数据
我们还利用合成数据生成流程来生成约 2 万亿个合成数据令牌。
包括合成数据生成流程在内的完整 recipe 将很快合并到
NVIDIA/NeMo-Curator
GitHub 存储库中。如要接收更新，请将存储库标记为星。
结束语
Nemotron-CC
是一个开放、大型、高质量的英语 Common Crawl 数据集，支持在短标记和长标记范围内预训练高度准确的 LLM。未来，我们希望发布更多数据集，这些数据集是先进 LLM 预训练的关键要素，例如专门的数学预训练数据集。
下载数据集
Common Crawl 中。
使用
NeMo 策展人
整理自己的数据集。
详细了解“Nemotron-CC：将 Common Crawl 转换为经过优化的长地平线预训练数据集”(Nemotron – CC：Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset) 中的技术细节。
致谢
我们感谢 Common Crawl Foundation 托管此数据集。我们感谢 Pedro Ortiz Suarez 为改进论文提供的宝贵反馈，以及 Greg Lindahl 为改进数据格式和布局提供的帮助。