hatemestinbejaia/mmarco-Arabic-mMiniLML-bi-encoder-KD-v1-Nonormalisation

SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hatemestinbejaia/mmarco-Arabic-mMiniLML-bi-encoder-KD-v1-Nonormalisation")
# Run inference
sentences = [
    'تحديد المسح',
    'المسح أو مسح الأراضي هو تقنية ومهنة وعلم تحديد المواقع الأرضية أو ثلاثية الأبعاد للنقاط والمسافات والزوايا بينها . يطلق على أخصائي مسح الأراضي اسم مساح الأراضي .',
    'إجمالي المحطات . تعد المحطات الإجمالية واحدة من أكثر أدوات المسح شيوعا المستخدمة اليوم . وهي تتألف من جهاز ثيودوليت إلكتروني ومكون إلكتروني لقياس المسافة ( EDM ) . تتوفر أيضا محطات روبوتية كاملة تتيح التشغيل لشخص واحد من خلال التحكم في الجهاز باستخدام جهاز التحكم عن بعد . تاريخ',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Reranking

• Evaluated with RerankingEvaluator

Metric	Value
map	0.5526
mrr@10	0.5566
ndcg@10	0.6272

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• learning_rate: 7e-05
• warmup_ratio: 0.07
• fp16: True
• half_precision_backend: amp
• load_best_model_at_end: True
• fp16_backend: amp

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 7e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.07
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: amp
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: amp
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	map
0.0512	2000	60.0645	39.7459	0.4997
0.1024	4000	38.1556	34.3446	0.4994
0.1536	6000	33.7868	32.9171	0.5159
0.2048	8000	31.8491	29.9714	0.5282
0.2560	10000	29.7765	29.9015	0.5078
0.3072	12000	27.5914	26.7202	0.5283
0.3584	14000	25.8129	25.0254	0.5430
0.4096	16000	24.0781	25.0622	0.5207
0.4608	18000	22.9328	23.7991	0.5433
0.5120	20000	21.7429	22.0272	0.5333
0.5632	22000	20.9529	20.9957	0.5485
0.6144	24000	19.9476	19.8111	0.5304
0.6656	26000	19.1556	19.2983	0.5363
0.7168	28000	18.5506	20.4461	0.5421
0.7680	30000	17.8418	19.6846	0.5192
0.8192	32000	17.4182	18.3179	0.5268
0.8704	34000	16.8575	18.5912	0.5401
0.9216	36000	16.4331	17.6217	0.5448
0.9728	38000	15.8319	16.4225	0.5469
1.0240	40000	14.5094	16.8592	0.5283
1.0752	42000	13.2263	15.6646	0.5511
1.1264	44000	12.9718	16.8053	0.5599
1.1776	46000	12.9135	16.9315	0.5557
1.2288	48000	12.6887	16.6569	0.5588
1.2800	50000	12.4705	15.5349	0.5569
1.3312	52000	12.3431	15.9067	0.5597
1.3824	54000	12.0741	15.0079	0.5668
1.4336	56000	11.9194	14.9333	0.5532
1.4848	58000	11.7261	14.3567	0.5598
1.5360	60000	11.5138	14.8380	0.5608
1.5872	62000	11.3494	13.7454	0.5544
1.6384	64000	11.116	14.3529	0.5527
1.6896	66000	11.0054	13.8486	0.5403
1.7408	68000	10.8677	13.8550	0.5598
1.7920	70000	10.6486	15.1113	0.5526
1.8432	72000	10.4977	13.7056	0.5580
1.8944	74000	10.3649	14.4802	0.5526

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.4.1+cu121
• Accelerate: 1.4.0
• Datasets: 3.2.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}