fineweb-nemotron-edu-score

This model is a fine-tuned version of intfloat/multilingual-e5-base on an transferred from English dataset. It achieves the following results on the evaluation set:

• Loss: 0.0489
• Precision: 0.9571
• Recall: 0.9605
• F1 Macro: 0.9588
• Accuracy: 0.9632

Model description

This model measure educational value of the given text for humans, as labelled by Nemotron-340B model.

Intended uses & limitations

Data filtering and evaluation of pretraining data at scale.

Training and evaluation data

Take a look at https://github.com/lapa-llm/lapa-llm/blob/main/pretraining/quality-classifiers/fineweb_hf.py

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 128
• seed: 0
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 256
• total_eval_batch_size: 1024
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	1.0257	0.3334	0.5	0.4001	0.6668
0.0562	1.3793	200	0.0520	0.9537	0.9549	0.9543	0.9593
0.0517	2.7586	400	0.0489	0.9571	0.9605	0.9588	0.9632

Framework versions

• Transformers 4.56.1
• Pytorch 2.6.0a0+ecf3bae40a.nv25.01
• Datasets 4.0.0
• Tokenizers 0.22.0