alignment-score-model

This model is a fine-tuned version of intfloat/multilingual-e5-base on the alignment dataset. It achieves the following results on the evaluation set:

• Loss: 0.0610
• Precision: 0.9802
• Recall: 0.9786
• F1 Macro: 0.9790
• Accuracy: 0.9790

Training script is available here: https://github.com/lapa-llm/lapa-llm/blob/main/pretraining/quality-classifiers/alignment_score.py

Model description

This model measure how likely the given text is a disinformation or unaligned to Ukrainian context.

Intended uses & limitations

Data filtering and evaluation of pretraining data at scale

Training and evaluation data

Take a look into https://github.com/lapa-llm/lapa-llm/blob/main/pretraining/quality-classifiers/alignment_score.py

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-06
• train_batch_size: 32
• eval_batch_size: 128
• seed: 0
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 256
• total_eval_batch_size: 1024
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 40
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	0.3688	0.2550	0.5	0.3377	0.5100
No log	1.0	31	0.2516	0.7558	0.5033	0.3450	0.5132
No log	2.0	62	0.1391	0.8851	0.8467	0.8454	0.8498
No log	3.0	93	0.1016	0.9340	0.9209	0.9217	0.9225
0.1646	4.0	124	0.0770	0.9693	0.9659	0.9665	0.9666
0.1646	5.0	155	0.0648	0.9778	0.9758	0.9763	0.9763
0.1646	6.0	186	0.0610	0.9802	0.9786	0.9790	0.9790

Framework versions

• Transformers 4.56.1
• Pytorch 2.6.0a0+ecf3bae40a.nv25.01
• Datasets 4.0.0
• Tokenizers 0.22.0