LamaDiab/MiniLM-V26Data-256HardBATCH-SemanticEngine

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("LamaDiab/MiniLM-V26Data-256HardBATCH-SemanticEngine")
# Run inference
sentences = [
    'diary of a wimpy kid do-it-youself book',
    'children english book',
    'grand theft auto v (ps5)',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5029, 0.2692],
#         [0.5029, 1.0000, 0.3907],
#         [0.2692, 0.3907, 1.0000]])

Evaluation

Metrics

Triplet

• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9444

Training Details

Training Dataset

Unnamed Dataset

• Size: 604,740 training samples
• Columns: anchor, positive, and itemCategory
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	itemCategory
  type	string	string	string
  details	min: 3 tokens mean: 9.9 tokens max: 42 tokens	min: 3 tokens mean: 4.76 tokens max: 78 tokens	min: 3 tokens mean: 3.94 tokens max: 9 tokens

• Samples:

  anchor	positive	itemCategory
  oval silver bracelet with side oat	women bracelet	bracelet
  petrol denim bandana	hair accessory	hair accessory
  lights & colors driver	toy	toddler toy

• Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 9,509 evaluation samples
• Columns: anchor, positive, negative, and itemCategory
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative	itemCategory
  type	string	string	string	string
  details	min: 3 tokens mean: 9.63 tokens max: 43 tokens	min: 2 tokens mean: 6.39 tokens max: 150 tokens	min: 3 tokens mean: 9.42 tokens max: 46 tokens	min: 3 tokens mean: 3.88 tokens max: 10 tokens

• Samples:

  anchor	positive	negative	itemCategory
  pilot mechanical pencil progrex h-127 - 0.7 mm	office supplies	artist pen extra fine brush tip b no.125	pencil
  superior drawing marker -pen - set of 12 colors - 2 nib	marker pen	urban stars prayer mat	marker
  first person singular author: haruki murakami	haruki murakami book	colorful rings arrangement game	literature and fiction

• Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 3e-05
• weight_decay: 0.01
• warmup_ratio: 0.1
• fp16: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 2
• dataloader_persistent_workers: True
• push_to_hub: True
• hub_model_id: LamaDiab/MiniLM-V26Data-256HardBATCH-SemanticEngine
• hub_strategy: all_checkpoints

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 3e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 2
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: True
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: LamaDiab/MiniLM-V26Data-256HardBATCH-SemanticEngine
• hub_strategy: all_checkpoints
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	cosine_accuracy
0.0004	1	4.1437	-	-
0.4232	1000	3.6457	0.9829	0.9363
0.8464	2000	2.6898	0.9383	0.9378
1.2693	3000	2.4448	0.9807	0.9374
1.6922	4000	2.6158	0.9553	0.9423
2.1150	5000	2.5157	0.9650	0.9416
2.5378	6000	2.3962	0.9475	0.9438
2.9607	7000	2.4768	0.9481	0.9444

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.1.2
• Transformers: 4.53.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.9.0
• Datasets: 4.4.1
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}