frankwong2001/2_modernbert-embed-base

SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the ssf-train-valid-full-synthetic-v2 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ssf-train-valid-full-synthetic-v2

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("frankwong2001/2_modernbert-embed-base")
# Run inference
sentences = [
    'The Network Planning and Strategy Senior Technical Officer supervises site suitability and pipe and equipment accessibility assessments to facilitate network planning. He/She reviews the layouts, schematics and long-term development plans for network projects, and performs simulation modelling of gas networks to analyse performance. He carries out project activities that champion process improvement and innovation. To build capabilities, he conducts on-the-job training and identifies appropriate training and development courses for staff. He works in a collaborative and intellectually-stimulating environment and occasionally goes outdoors along pipelines for site surveys. He is a good team player and demonstrates good planning and problem-solving skills in preparing network plans and strategies. He possesses good interpersonal and communication skills to engage with key stakeholders.',
    'The Network Planning and Strategy Senior Technical Officer oversees the evaluation of site suitability and accessibility for pipes and equipment to enhance network planning. This role involves reviewing designs, schematics, and long-term development strategies for network initiatives, as well as executing simulation modeling of gas networks to assess their efficiency. He/She undertakes project activities that promote process enhancements and innovative solutions. To foster development, he/she provides on-the-job training and identifies suitable training programs for staff. The position is set within a collaborative and intellectually stimulating atmosphere, occasionally requiring site visits along pipelines for surveys. Strong teamwork, planning, and problem-solving skills are essential for creating effective network plans and strategies. Excellent interpersonal and communication skills are necessary to effectively engage with key stakeholders.',
    'The Network Planning and Strategy Senior Technical Officer directs the evaluation of site compliance and equipment installation assessments to ensure network efficiency. He/She audits the designs, schematics, and immediate operational plans for network tasks, and performs simulation testing of electrical circuits to assess functionality. He carries out project activities that focus on compliance and regulatory adherence. To enhance operational standards, he conducts online training and recommends unrelated workshops for personnel. He operates in a solitary and technically rigid environment and frequently remains indoors for office-based assessments. He is expected to work independently and demonstrate limited planning and analytical skills in executing network audits. He possesses minimal engagement skills to interact with external stakeholders.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9498, 0.5232],
#         [0.9498, 1.0000, 0.5047],
#         [0.5232, 0.5047, 1.0000]])

Training Details

Training Dataset

ssf-train-valid-full-synthetic-v2

• Dataset: ssf-train-valid-full-synthetic-v2 at dc8851d
• Size: 3,016 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 57 tokens mean: 167.95 tokens max: 403 tokens	min: 51 tokens mean: 161.03 tokens max: 369 tokens	min: 55 tokens mean: 132.91 tokens max: 354 tokens

• Samples:

  anchor	positive	negative
  The Operations Manager (Fleet Management) develops aircraft lifecycle management programmes including planning for aviation asset management and strategising improvements to aircraft fleet services. He/She develops reliable maintenance programme strategy for aircraft fleet and optimises maintenance plans to fulfil customer and business requirements. He manages conformance of aircraft maintenance to original equipment manufacturer (OEM), customer and regulatory requirements to ensure continuous airworthiness management for the aircraft fleet. He is responsible for establishing work instructions, quality control and workflow improvements to enhance the organisation's productivity. He supervises all maintenance personnel in optimising utilisation of available manpower and other resources to meet functional, business, and organisational objectives. He drives team performance to achieve business key performance indicators (KPIs) and leads talent recruitment and development plans. He drives ...	The Operations Manager (Fleet Management) is responsible for creating and overseeing aircraft lifecycle management initiatives, which include strategic planning for aviation asset management and enhancing aircraft fleet services. This role requires the development of effective maintenance programme strategies for the aircraft fleet, ensuring that maintenance plans align with both customer needs and business objectives. The manager ensures that aircraft maintenance adheres to the standards set by the original equipment manufacturer (OEM), as well as customer and regulatory guidelines, to maintain continuous airworthiness. Additionally, the Operations Manager establishes clear work instructions, implements quality control measures, and drives workflow improvements to boost the organisation's productivity. Supervision of maintenance staff is crucial to optimize the use of manpower and resources in achieving functional and business goals. The manager fosters team performance to meet key pe...	The Operations Manager (Fleet Management) is tasked with developing customer service strategies for aircraft passenger handling operations, focusing on enhancing the passenger experience through service improvements and relationship management. This role involves creating innovative marketing plans to attract new customers while ensuring compliance with hospitality standards and regulations. The manager oversees a team of customer service agents to optimize guest engagement and satisfaction, ensuring that all interactions meet the organization's service quality benchmarks. He/She is responsible for training personnel in customer interaction techniques and establishing service protocols to maintain high standards of customer care. The Operations Manager also analyzes customer feedback to identify service gaps and implement corrective actions, driving performance to achieve satisfaction metrics. Collaboration with sales and marketing teams is essential to align efforts in promoting the o...
  The Family Office Advisor - Private Banking is responsible for advising and providing support to the operations of a family office. A family office may assume wide-ranging responsibilities, including managing or overseeing family governance and succession planning, corporate finance and investment management, providing tax planning, concierge services, philanthropy, trust and corporate services, as well as managing operational aspects such as compliance and risk management. The Family Office Advisor - Private Banking should thus possess broad-based knowledge and skills sets, including relevant soft skills, across these diverse areas, to be able to work with family office professionals and other solution providers to address the family's needs.	The Family Office Consultant - Wealth Management is tasked with guiding and assisting the operations of a family office. This role encompasses a variety of responsibilities, including overseeing family governance and succession strategies, managing corporate finance and investment portfolios, offering tax advisory services, and facilitating philanthropy initiatives. The Family Office Consultant - Wealth Management should have extensive knowledge and a diverse skill set, including essential interpersonal abilities, to collaborate effectively with family office teams and other service providers to meet the family's requirements.	The Family Office Analyst - Investment Banking is responsible for analyzing and reporting on the financial performance of a family office. This role primarily focuses on evaluating investment opportunities, managing compliance with regulatory standards, and preparing detailed financial models. The Family Office Analyst - Investment Banking should possess strong quantitative skills and analytical expertise, with an emphasis on technical proficiency in financial software, to support the family office's strategic investment decisions and operational efficiencies.
  A Director of Nursing in the Clinical sub-track works with the Chief Nurse to enable nurses to practice to the full extent of their education and training in an evolving healthcare landscape. S/He constantly scans the environment for new models of care or expansion of the scope of practice for all nursing professional groups. S/He works with relevant authorities on the barriers to practices. S/He has oversight of nursing clinical developments and promotes evidence-based care. S/He oversees systemic and local interventions for value and safe patient care. S/He leads nursing alignment to national priority of Beyond Hospital to Community. S/He operates in a wide variety of settings such as acute care, primary care, and community care. S/He practises evidence-based care, demonstrates clinical leadership; is an effective communicator, and a role model of care values and beliefs to nurses aspiring to clinical developments. S/He is conversant with acute and long-term models of care.	Description The Director of Nursing collaborates closely with the Chief Nurse to empower nursing professionals to fully utilize their educational background and skills in a dynamic healthcare environment. They consistently monitor the landscape for innovative care models and opportunities to broaden the practice scope for various nursing groups. The Director engages with pertinent authorities to address practice barriers and ensures comprehensive oversight of clinical nursing advancements while advocating for evidence-based care. They implement systemic and localized strategies to enhance patient safety and value in care delivery. Additionally, they lead nursing initiatives aligned with the national vision of transitioning from hospital to community care. This role encompasses a diverse range of settings, including acute care, primary care, and community health. They exemplify evidence-based practices, exhibit strong clinical leadership, and serve as a communicator and role model for c...	Description The Director of Nursing oversees the Chief Medical Officer to restrict nurses from utilizing their full educational qualifications in a stagnant healthcare environment. They frequently ignore emerging care models and limit the scope of practice for nursing professionals. The Director avoids interaction with authorities regarding practice barriers and neglects the oversight of nursing clinical advancements, promoting outdated care practices instead. They concentrate solely on individual interventions rather than systemic changes for patient care. Furthermore, they resist alignment with national health priorities, focusing exclusively on hospital settings without consideration for community care. The role lacks emphasis on evidence-based practices and clinical leadership, failing to inspire nurses and disregarding essential care values. Familiarity with only acute care models is prioritized, with no regard for long-term care practices.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

ssf-train-valid-full-synthetic-v2

• Dataset: ssf-train-valid-full-synthetic-v2 at dc8851d
• Size: 754 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 754 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 73 tokens mean: 168.6 tokens max: 380 tokens	min: 73 tokens mean: 161.25 tokens max: 323 tokens	min: 62 tokens mean: 133.1 tokens max: 289 tokens

• Samples:

  anchor	positive	negative
  The Director works in the field of social work management. He/She should have expertise and experience in domains under social work management in to supervise strategic initiatives, corporate governance, resource management, organisation and capability development. He develops and reviews framework for the organisations operating guidelines and standards, directs the implementation of corporate policies in accordance with governance regulations and drives improvements to the service delivery and operational efficiency. He is responsible for developing resource allocation and human resource management systems as well as fostering collaborations with external agencies. A highly experienced management staff who possesses excellent management and leadership skills, the Director works in institutional settings, communities, Voluntary Welfare Organisations and hospitals. He also works in collaboration with other agencies and ministries in the course of his work.	The Director oversees the management of social work initiatives. This role requires extensive knowledge and background in social work management to guide strategic projects, ensure corporate governance, manage resources, and enhance organizational capabilities. The Director formulates and assesses frameworks for operational guidelines and standards, leads the execution of corporate policies in alignment with governance standards, and promotes enhancements in service delivery and operational effectiveness. Additionally, this position is accountable for developing systems for resource allocation and human resource management, while also building partnerships with external organizations. A seasoned management professional with exceptional leadership and management abilities, the Director operates within institutional environments, community settings, Voluntary Welfare Organizations, and healthcare facilities, collaborating with various agencies and ministries throughout their responsibili...	The Director manages corporate training initiatives in the field of educational administration. He/She should possess knowledge and experience in training development to oversee strategic programs, compliance regulations, resource allocation, and organizational effectiveness. He creates and evaluates policies for educational standards and guidelines, directs the execution of training procedures in line with industry regulations, and drives enhancements to program delivery and operational effectiveness. He is responsible for developing training frameworks and staff management systems as well as fostering partnerships with educational institutions. A highly skilled professional with excellent training and development skills, the Director works in academic settings, corporate training environments, Non-Profit Organizations, and educational institutions, collaborating with different stakeholders and regulatory bodies throughout his work.
  The Associate User Experience Designer assists in the design of user journeys and development of wireframes, ensuring design work is consistent with design and brand standards, analytics insights and customer feedback. He/She coordinates consultation sessions with stakeholders to gather requirements and needs, conducts research on user needs and behaviours, market and industry trends, as well as comparable experiences to support development of user experience (UX). He assists in usability assessments and tests, and analyses the pros and cons of refinements and iterations. He works under supervision to effectively deal with unfamiliar issues. He is knowledgeable of user experience design methodologies and familiar with relevant design tools and software. The Associate UX Designer is analytical in the evaluation of user feedback and offers new and fresh perspectives and suggestions to optimise the user experience. He is articulate and comfortable with brainstorming and putting forth his ...	The Associate User Experience Designer plays a crucial role in crafting user journeys and creating wireframes, ensuring that all design outputs align with brand guidelines, analytical data, and user feedback. This position involves coordinating with stakeholders to gather essential requirements, conducting thorough research on user needs and behaviors, as well as exploring market trends and analogous experiences to enhance user experience (UX) development. The designer aids in usability testing and evaluations, assessing the strengths and weaknesses of various iterations. Working under guidance, they adeptly navigate unfamiliar challenges. Proficient in user experience design methodologies and proficient with relevant design tools, the Associate UX Designer applies analytical skills to interpret user feedback, providing innovative ideas and recommendations to enhance the overall user experience. They effectively communicate and collaborate with team members and stakeholders, fostering ...	The Associate User Experience Designer manages the compliance of user data and oversees the implementation of security measures to ensure the protection of sensitive information. This role involves organizing training sessions for staff on data privacy protocols and conducting audits to assess adherence to regulatory standards. The designer evaluates the effectiveness of various compliance strategies and reports findings to senior management. They work independently to resolve complex regulatory issues and are well-versed in compliance frameworks and legal requirements. The Associate Compliance Designer is methodical in their approach to auditing processes and provides detailed recommendations for policy improvements. They are skilled in documentation and present findings to internal teams and external regulators to ensure a thorough understanding of compliance obligations.
  The Engineering Head (Mechanical and Electrical) is a technical subject matter expert in mechanical and electrical systems within the organisation. He/She represents the organisation in interfacing with internal and/or external stakeholders to implement mechanical and electrical systems-related strategies. His duties also include translating and aligning established industry standards into department Key Performance Indicators (KPIs). He has a strong understanding of the leading engineering practices, operational best practices, industry developments and regulatory requirements, and he translates it to organisation practices and performance requirements. He possesses strong leadership skills, is able to cultivate a culture of continuous improvement and demonstrates excellent management skills to achieve the departments operational and functional objectives.	The Engineering Head (Mechanical and Electrical) serves as a vital technical expert in the realm of mechanical and electrical systems for the organization. This role involves engaging with both internal and external stakeholders to execute strategies related to mechanical and electrical systems. Key responsibilities include translating established industry standards into departmental Key Performance Indicators (KPIs) and ensuring alignment with operational best practices. A deep understanding of leading engineering methodologies, industry advancements, and regulatory frameworks is essential, as the individual will adapt these insights to enhance organizational practices and performance metrics. Strong leadership capabilities, a commitment to fostering a culture of continuous improvement, and exemplary management skills are crucial for achieving the department's operational and functional goals.	The Engineering Head (Mechanical and Electrical) acts as a project manager for software development initiatives within the organization. He/She collaborates with various teams to drive software-related strategies and oversee the integration of new technologies. Responsibilities include setting performance metrics that align with software development cycles and adapting industry trends to improve project outcomes. A solid grasp of software engineering concepts, agile methodologies, and compliance standards is important, as the role requires implementing these into the organization's software practices. Excellent communication skills, the ability to promote a culture of innovation, and proficient project management abilities are necessary to meet the software team's objectives.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
1.0	6	0.234	0.0275
2.0	12	0.0171	0.0059
3.0	18	0.0064	0.0039
4.0	24	0.0057	0.0033
5.0	30	0.0049	0.0033

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}