frankwong2001/3_modernbert-embed-base

SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the ssf-train-valid-full-synthetic-v3 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ssf-train-valid-full-synthetic-v3

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("frankwong2001/3_modernbert-embed-base")
# Run inference
sentences = [
    'The Principal Security Consultant is responsible for leading a team to clinch consultancy projects to provide security audits, reviews and security risk assessment services to clients and recommend improvements to existing security measures. He/She is required to evaluate tender documents and manage the deployment of security consultants to develop security protection and implementation plans for various types of facilities. He is required to work in an office environment and perform site visits when necessary. He is expected to communicate with relevant stakeholders and clients as part of his role in performing the respective duties. This requires him to be analytical, responsive, decisive and cooperative.',
    'The Principal Security Consultant is tasked with spearheading a team to secure consultancy projects focused on delivering security audits, assessments, and risk evaluations to clients, while also suggesting enhancements to current security protocols. This role involves reviewing tender documents and overseeing the assignment of security specialists to formulate protection strategies and implementation plans for diverse facilities. The position primarily operates in an office setting, with occasional site visits required. Effective communication with stakeholders and clients is essential for fulfilling the responsibilities of this role, demanding strong analytical, responsive, decisive, and collaborative skills.',
    'The Principal Security Consultant is responsible for managing a team to oversee marketing campaigns, analyzing market trends, and developing promotional strategies for various products. \n\nThe Principal Security Consultant oversees the operations of a junior audit team, focusing on compliance with internal policies and ensuring adherence to basic regulatory requirements.\n\nThe Principal Security Consultant is engaged in financial analysis, requiring expertise in budget forecasting and revenue management within the retail sector.\n\nThe Principal Security Consultant is tasked with leading cross-border legal compliance initiatives, requiring knowledge of international law and regulations specific to the European market.\n\nThe Principal Security Consultant combines project management and IT support roles, focusing on both strategic planning and technical troubleshooting for software applications.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9259, 0.2866],
#         [0.9259, 1.0000, 0.3057],
#         [0.2866, 0.3057, 1.0000]])

Training Details

Training Dataset

ssf-train-valid-full-synthetic-v3

• Dataset: ssf-train-valid-full-synthetic-v3 at b816c6b
• Size: 3,016 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 66 tokens mean: 169.89 tokens max: 355 tokens	min: 67 tokens mean: 160.73 tokens max: 323 tokens	min: 80 tokens mean: 296.5 tokens max: 1718 tokens

• Samples:

  anchor	positive	negative
  The Branch Manager is responsible for ensuring the achievement of the branch's financial targets. He/She is responsible for all functions of the branch under his care, such as hiring employees, implementing service initiatives, overseeing the approval of loans and lines of credit, marketing, and assisting with customer relations. He effectively manages team members within his branch, including developing and motivating them to perform and achieve sales targets. The Branch Manager may occasionally be required to work weekends and after hours. He has good organisational skills, is diligent and possesses strong people management capabilities. He is customer focused, has keen problem solving abilities and is able to manage internal and external stakeholders from a variety of backgrounds.	The Branch Manager oversees the successful attainment of the branch's financial objectives. This role encompasses all operational aspects of the branch, including recruiting staff, executing service strategies, managing loan and credit approvals, conducting marketing efforts, and facilitating customer engagement. The Branch Manager adeptly leads team members, fostering their development and driving them towards achieving sales goals. Occasionally, this position may require working on weekends and after hours. The ideal candidate demonstrates excellent organizational skills, a strong work ethic, and possesses exceptional people management abilities. A customer-centric approach, along with proficient problem-solving skills and the ability to engage with diverse internal and external stakeholders, is essential.	The Branch Manager is tasked with ensuring compliance with environmental regulations across various departments. He/She is responsible for all aspects of environmental audits, including recruiting compliance officers, implementing sustainability initiatives, overseeing the approval of waste disposal methods, marketing green practices, and managing community relations. He effectively collaborates with team members to enhance compliance metrics and achieve sustainability targets. The Branch Manager may be required to work in extreme weather conditions. He has strong analytical skills, is detail-oriented, and possesses excellent project management capabilities. He is focused on environmental impact, has sharp analytical abilities, and can manage regulatory stakeholders from different sectors. The Branch Manager is accountable for the strategic growth of multiple branches within an international organization. This senior role involves high-level decision-making, including mergers and acqu...
  The Head of IT Audit develops the organisation's IT audit framework to manage regulatory and operational risks to safeguard IT assets. He/She defines key objectives and guiding principles for the formulation of IT risk management programs, as well as procedures for documenting and updating policies, standards, guidelines relating to the management of IT assets. He advices on the development of IT audit plans and ensures that audit plans comply with regulatory, operational, security risks and relevant internal auditing standards. He oversees the conduct of audits, respective investigations into non-compliance and risks identified from audits. He overlooks new IT policies, systems and processes necessary for enhancing IT controls and mitigate risks. He consults with and advises senior leaders regarding internal controls and security procedures, prepares activity and progress reports relating to the IT audit function. He also guide team members on procedures, technical problems, prioritie...	The Head of IT Audit is responsible for establishing a comprehensive IT audit framework that effectively manages regulatory and operational risks to protect the organization’s IT assets. This role involves defining essential objectives and principles for developing IT risk management strategies, as well as creating procedures for the documentation and updating of policies, standards, and guidelines related to IT asset management. The individual provides strategic advice on IT audit planning and ensures compliance with relevant regulations, operational standards, and security risks in line with internal auditing practices. Additionally, the Head of IT Audit supervises audit execution, investigates instances of non-compliance, and addresses risks identified during audits. The role includes overseeing the implementation of new IT policies, systems, and processes to strengthen IT controls and mitigate risks. The Head of IT Audit collaborates with senior leadership to enhance internal contr...	The Head of IT Audit is tasked with managing financial audits across various departments to ensure compliance with budgetary constraints and fiscal regulations. The Head of IT Audit functions as a Junior Audit Associate, responsible for assisting in the execution of basic audit tasks under close supervision, with limited decision-making authority. The Head of IT Audit serves as a Compliance Officer in the healthcare sector, focusing on regulatory compliance and quality assurance in clinical practices rather than IT environments. The Head of IT Audit oversees cross-border financial reporting for international subsidiaries, navigating different accounting standards and regulations while ensuring compliance with local market practices. The Head of IT Audit combines responsibilities of a Project Manager and a Risk Analyst, overseeing multiple unrelated projects while simultaneously assessing risks across diverse business units without a clear focus on IT auditing.
  A Nurse Manager is responsible for planning, coordinating, directing, and evaluating operational activities and resource utilisation in the department. S/He is also responsible for managing nursing manpower operating expenses and budget effectively to provide high quality patient care. S/He oversees at least one unit. S/He oversees the professional and personal development of all staff under her/his charge. Her/His core function is in managerial tasks, but s/he will also perform some clinical, educational and research tasks in the course of her/his day-to-day work. S/He provides guidance to assistant nurse clinicians and below to ensure optimal care is provided to meet desired patient outcomes and experience. S/He operates in a wide variety of settings such as acute care, primary care, community hospitals, integrated care and long-term care facilities. S/He should be resourceful, prudent, tactful and persuasive.	The Nurse Manager is tasked with organizing, supervising, and assessing the operational functions and resource allocation within the department. They are accountable for effectively managing nursing workforce budgets and expenses to ensure the delivery of exemplary patient care. The Nurse Manager oversees at least one nursing unit and is responsible for fostering both the professional and personal growth of all team members under their leadership. While primarily focused on managerial duties, they also engage in clinical, educational, and research activities as part of their daily responsibilities. Additionally, they provide support and direction to assistant nurse clinicians and lower-level staff to guarantee that optimal care is provided, aligning with desired patient outcomes and experiences. The role is carried out in diverse environments, including acute care, primary care, community hospitals, integrated care, and long-term care facilities. Ideal candidates will be resourceful, p...	The Nurse Manager is focused on developing marketing strategies, coordinating promotional activities, and evaluating the effectiveness of advertising campaigns in the department. They are responsible for managing the budget for marketing materials and operational expenses to enhance brand visibility. The Nurse Manager oversees at least one team of marketing specialists and is tasked with driving the professional growth of all personnel involved. While their primary role is in marketing tasks, they will also conduct some sales training and customer research during their daily activities. They provide mentorship to junior marketing associates and below to ensure effective communication is maintained with clients. The position is performed in various sectors, such as retail, hospitality, and entertainment. Candidates should be innovative, analytical, strategic, and persuasive. The Nurse Manager is responsible for supervising nursing operations at a senior level, managing large-scale budg...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

ssf-train-valid-full-synthetic-v3

• Dataset: ssf-train-valid-full-synthetic-v3 at b816c6b
• Size: 754 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 754 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 58 tokens mean: 169.85 tokens max: 380 tokens	min: 54 tokens mean: 160.26 tokens max: 362 tokens	min: 94 tokens mean: 304.69 tokens max: 1330 tokens

• Samples:

  anchor	positive	negative
  The Venue Operations Manager is responsible for overseeing the running of venue operations, including the logistics requirements. He/She works closely with event services department to ensure client requirements are fulfilled in compliance to local health and safety standards. He reviews event plans to ensure generation of maximum yield for organisation. Meticulous and resourceful, he possesses excellent problem-solving skills and is able to react quickly to deviations in the project plans. He is able to work in a flexible workweek, including weekends, evenings, and public holidays, and is comfortable working in both an indoor and outdoor environment depending on the nature and requirements of the events.	The Venue Operations Manager is tasked with managing the day-to-day functions of venue operations, including logistical planning and execution. Collaborating closely with the event services team, he/she ensures that client needs are met while adhering to local health and safety regulations. The manager evaluates event proposals to maximize organizational profitability. Detail-oriented and innovative, he/she demonstrates strong problem-solving abilities and can swiftly adjust to changes in project timelines. Flexibility in working hours is essential, as the role may require availability during weekends, evenings, and public holidays, and the manager must be adept at working in various environments, whether indoor or outdoor, based on event specifications.	The Venue Operations Coordinator is involved in managing the administrative tasks of venue bookings, focusing on client communications rather than logistics. The Venue Operations Director oversees large-scale venue strategies and has decision-making authority over multiple departments, requiring extensive experience in leadership roles. The Venue Operations Specialist is responsible for compliance checks in a manufacturing facility, utilizing similar analytical skills but within a different regulatory framework. The Venue Operations Manager in a different country is tasked with adhering to international safety standards while managing events that cater to a diverse clientele across borders. The Venue Operations Manager combines responsibilities of a catering manager and a maintenance supervisor, overseeing food services while also handling facility repairs, which creates confusion in role expectations.
  The Network Development Principal Engineer provides technical leadership to the network development team and develops detailed project plans for electricity transmission and/or distribution network development and/or the integration of distributed generation sources and energy storage systems with the grid. As the technical expert, he/she reviews project progress reports and investigation findings of site problems encountered to propose follow- up actions. He reviews installation plans for metering equipment and sensors, and leads process improvement initiatives. He leads technical capability development programmes, including on-the-job training and coaching, and formulates the technical training and development plans for the teams. He manages the Permits-to-Work for the team, and establishes Safe System of Work (SSoW) frameworks and practices for his area of work. He proposes emergency technical and recovery activities based on the crisis management framework, and determines the respo...	The Network Development Principal Engineer offers expert technical guidance to the network development team, crafting comprehensive project plans for the enhancement of electricity transmission and distribution networks, as well as the incorporation of distributed energy resources and storage systems into the grid. Acting as the technical authority, he/she evaluates project progress reports and conducts investigations into site issues, recommending appropriate follow-up actions. He/she assesses installation strategies for metering devices and sensors, spearheads initiatives for process enhancements, and directs technical capability-building programs, including hands-on training and mentorship, while devising technical training and development strategies for the team. Additionally, he/she oversees the management of Permits-to-Work for the team, implementing Safe System of Work (SSoW) frameworks and practices specific to the area of responsibility. He/she formulates emergency technical a...	The Network Development Principal Engineer manages the financial operations of the network development team, focusing on budgeting and expense tracking for electricity transmission projects. The Network Development Principal Engineer serves as a Junior Engineer who assists with basic tasks under close supervision, requiring minimal experience in network development and no decision-making authority. The Network Development Principal Engineer operates as a Compliance Associate in the healthcare sector, ensuring that all regulatory requirements are met while analyzing data for compliance with health regulations. The Network Development Principal Engineer focuses on network development projects in a different regulatory environment, specifically in the European market, adapting to varied local standards and practices. The Network Development Principal Engineer combines the roles of a project manager and a sales representative, overseeing project timelines while also directly selling n...
  The Keeper/Aquarist is responsible for the care and management of wildlife within the organisation/attractions sites. This includes the preparation and feeding of food to the wildlife, caring for ill animals, checking enclosures and cages for signs of wear or damage for animal, staff and visitor safety, and giving educational talks/tours to the visitors. He/she also maintains animal training behaviours and promotes conservation awareness through his animal presentations. Detail-oriented with a strong passion for animals, he is attentive to the needs of the wildlife under his care, and leverages his strong communication skills to communicate effectively with visitors on the characteristics and behaviours of the wildlife. He is physically fit and works in a shift system encompassing weekends and public holidays. Outside the working hours, he may be on a rota for call-outs. He often stays outdoors for long periods of time even through inclement weather, and may need a driving licence if h...	The Keeper/Aquarist is entrusted with the comprehensive care and management of wildlife within the organization’s attractions. This role involves preparing and providing nutritious food for the animals, tending to those that are unwell, and inspecting enclosures for any signs of damage to ensure the safety of animals, staff, and visitors. The Keeper/Aquarist also engages guests through educational talks and tours, enhancing their understanding of wildlife conservation. With a keen eye for detail and a deep passion for animals, he/she is dedicated to meeting the needs of the creatures in their care and utilizes strong communication skills to effectively share insights about animal behavior and characteristics with visitors. The position requires physical fitness, as it involves working shifts that include weekends and public holidays. Additionally, the Keeper/Aquarist may be on call outside regular hours and is accustomed to spending extended periods outdoors, regardless of weather cond...	The Keeper/Aquarist is responsible for managing the administrative tasks related to wildlife licensing and permits within the organization. This includes ensuring compliance with local regulations, filing necessary documents, and maintaining records, without direct interaction with the animals or visitors. The Keeper/Aquarist oversees a team of junior staff members and is focused on strategic planning for conservation policies rather than hands-on animal care. This role requires several years of experience in management and decision-making authority regarding budget allocations and resource management. The Keeper/Aquarist works as a Compliance Associate in the banking sector, where he/she analyzes regulatory frameworks and ensures adherence to financial policies. This position requires a strong understanding of financial compliance rather than animal welfare. The Keeper/Aquarist is tasked with wildlife management in an international context, focusing on cross-border regulatory issue...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
1.0	6	0.2184	0.0124
2.0	12	0.0066	0.0010
3.0	18	0.0031	0.0006
4.0	24	0.0027	0.0006
5.0	30	0.002	0.0005

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}