RikoteMaster/embedder-granite

SentenceTransformer based on ibm-granite/granite-embedding-107m-multilingual

This is a sentence-transformers model finetuned from ibm-granite/granite-embedding-107m-multilingual. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: ibm-granite/granite-embedding-107m-multilingual
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("RikoteMaster/embedder-granite")
# Run inference
sentences = [
    'Antiprotozoal 923 MEFLOQUINE Mefloquine is effective therapy of other Although toxicity is mefloquine one recommended for most regions with Chemistry Mefloquine is 4-quinoline methanol is chemically quinine. can given because local irritation with parenteral and hours. Mefloquine highly uted and treat- regimen. elimination half-life about 20 allowing dosing chemoprophylaxis. With dos- drug reached over number of interval can be shortened to 4 with daily doses 250 mg, this is not and metabolites of in can be in the months completion therapy. Antimalarial Action & strong P falciparum P is hepatic stages or gametocytes. The of unknown. Sporadic mefloquine been from areas. At resistance appears to uncommon regions Asia high rates border areas resis- tance quinine resistance to Clinical in',
    'CHAPTER 52 Antiprotozoal Drugs 923 MEFLOQUINE Mefloquine is effective therapy for many chloroquine-resistant strains of P falciparum and against other species. Although toxicity is a concern, mefloquine is one of the recommended chemopro- phylactic drugs for use in most malaria-endemic regions with chloroquine-resistant strains. Chemistry & Pharmacokinetics Mefloquine hydrochloride is a synthetic 4-quinoline methanol that is chemically related to quinine. It can only be given orally because severe local irritation occurs with parenteral use. It is well absorbed, and peak plasma concentrations are reached in about 18 hours. Mefloquine is highly protein-bound, extensively distrib- uted in tissues, and eliminated slowly, allowing a single-dose treat- ment regimen. The terminal elimination half-life is about 20 days, allowing weekly dosing for chemoprophylaxis. With weekly dos- ing, steady-state drug levels are reached over a number of weeks; this interval can be shortened to 4 days by beginning a course with three consecutive daily doses of 250 mg, although this is not stan- dard practice. Mefloquine and acid metabolites of the drug are slowly excreted, mainly in the feces. The drug can be detected in the blood for months after the completion of therapy. Antimalarial Action & Resistance Mefloquine has strong blood schizonticidal activity against P falciparum and P vivax, but it is not active against hepatic stages or gametocytes. The mechanism of action of mefloquine is unknown. Sporadic resistance to mefloquine has been reported from many areas. At present, resistance appears to be uncommon except in regions of Southeast Asia with high rates of multidrug resistance (especially border areas of Thailand). Mefloquine resis- tance appears to be associated with resistance to quinine and halofantrine but not with resistance to chloroquine. Clinical Uses A. Chemoprophylaxis Mefloquine is effective in prophylaxis against most strain',
    'the body to colonize various organs in the process called metastasis. Such tumor stem cells thus can express clonogenic (colony-forming) capability, and they are characterized by chromosome abnormalities reflecting their genetic instability, which leads to progressive selection of subclones that can survive more readily in the multicellular environment of the host. This genetic instability also allows them to become resistant to chemotherapy and radiotherapy. The invasive and metastatic processes as well as a series of metabolic abnormalities associated with the cancer result in tumor-related symptoms and eventual death of the patient unless the neoplasm can be eradicated with treatment. 54 CAUSES OF CANCER The incidence, geographic distribution, and behavior of specific types of cancer are related to multiple factors, including sex, age, race, genetic predisposition, and exposure to environmental car- cinogens. Of these factors, environmental exposure is probably most important. Exposure to ionizing radiation has been well documented as a significant risk factor for a number of cancers, including acute leukemias, thyroid cancer, breast cancer, lung cancer, soft tissue sarcoma, and basal cell and squamous cell skin cancers. Chemical carcinogens (particularly those in tobacco smoke) as well as azo dyes, aflatoxins, asbestos, benzene, and radon have all been well documented as leading to a wide range of human cancers. Several viruses have been implicated in the etiology of various human cancers. For example, hepatitis B and hepatitis C are asso- ciated with the development of hepatocellular cancer; HIV is associated with Hodgkin’s and non-Hodgkin’s lymphomas; human papillomavirus is associated with cervical cancer and head and neck cancer; and Ebstein-Barr virus is associated with nasopharyn- geal cancer. Expression of virus-induced neoplasia may also depend on additional host and environmental factors that modu- late the transformation process. Cellular genes are known that are homologous to the transforming genes of the retroviruses, a family',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 34,441 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 3 tokens mean: 99.93 tokens max: 255 tokens	min: 14 tokens mean: 245.16 tokens max: 512 tokens

• Samples:

  anchor	positive
  March Lecture Solving using by Svensson1 In this following: We describe Multiplicative Hedge) • We this method to solve is these lecture are on of “Lecture 11 of in 2015” written and Simon Rodriguez and on by Kaul that the lecture previous we to use the majority method order to fairly general with days N experts as For t . , gives advice: 2. advice the expert, of and the decides 4. observes suffers was majority parameterized by ε “learning rate”), now as follows: • each i weight initialized 1. are trustworthy the ning.) each t: • Predict based on w(t) After observing the vector, i expert the lecture we case = following any sequence of i of WM mistake	Advanced Algorithms March 22, 2022 Lecture 9: Solving LPs using Multiplicative Weights Notes by Ola Svensson1 In this lecture we do the following: • We describe the Multiplicative Weight Update (actually Hedge) method. • We then use this method to solve covering LPs. • This is a very fast and simple (i.e., very attractive) method for solving these LPs approximately. These lecture notes are partly based on an updated version of “Lecture 11 of Topics in TCS, 2015” that were written by Vincent Eggerling and Simon Rodriguez and on the lecture notes by Shiva Kaul that we used in the last lecture. 1 Recall last lecture In the previous lecture, we saw how to use the weighted majority method in order to fairly smartly follow the advice of experts. Recall that the general game-setting with T days and N experts was as follows: For t = 1, . . . , T: 1. Each expert i ∈[N] gives some advice: UP or DOWN 2. Aggregator (you) predicts, based on the advice of the expert, UP or DOWN. 3. Adversary, with k...
  Last ε The same proof the For duration expert i ∈[N], of WM mistakes ε) · (# i’s mistakes) + O(log(N)/ε) 1Disclaimer: notes They not been and may typos,	Last lecture we analyzed the case when ε = 1/2. The same proof gives the following Theorem 1 For any sequence of outcomes, duration T, and expert i ∈[N], # of WM mistakes ≤2(1 + ε) · (# of i’s mistakes) + O(log(N)/ε) . 1Disclaimer: These notes were written as notes for the lecturer. They have not been peer-reviewed and may contain inconsistent notation, typos, and omit citations of relevant works. 1
  [Sketch] The proof done by potential function: for each = 1, . , 1, Φ(t) = i We lower potential the mistakes of i. We it in of our mistakes. The weight of expert down by a −ε) i does. As weight is 1, Φ(T +1) = +1) ≥w(T +1) = (1 −ε)# of . Every the experts was (since majority weights are (1 −ε). that the factor every time Φ(T −ε/2)# mistakes = N −ε/2)# , equality used that = was initialized with a weight above bounds give us (1 mistakes ≤N · (1 of . sides, allowing for randomized strategies In the exercises, you proved that are instances for weighted This overcome this we allow random instead of always making prediction the to create A is often general is often good the of adversaries. Allowing randomized leads to following with T t . . ,	Proof [Sketch] The proof was done by defining a potential function: for each t = 1, . . . , T + 1, let Φ(t) = X i∈[N] w(t) i . We now lower bound the “final” potential Φ(T +1) using the number of mistakes of i. We then upper bound it in terms of our number of mistakes. Lower bound: The weight of expert i goes down by a factor (1 −ε) for each mistake i does. As the initial weight of i is 1, Φ(T +1) = X j∈[N] w(T +1) j ≥w(T +1) i = (1 −ε)# of i’s mistakes . Upper bound: Every time WM errs, at least half the weight of the experts was wrong (since weighted majority was wrong). These weights are then decreased by (1 −ε). It follows that the potential goes down by at least a factor (1 −ε/2) every time WM errs. And so Φ(T +1) ≤Φ(1) · (1 −ε/2)# of WM mistakes = N · (1 −ε/2)# of WM mistakes , where for the equality we used that Φ(1) = N since each expert was initialized with a weight of 1. The above bounds give us (1 −ε)# of i’s mistakes ≤Φ(T +1) ≤N · (1 −ε/2)# of WM mistakes . Taking logs on b...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 3,827 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 15 tokens mean: 174.64 tokens max: 266 tokens	min: 55 tokens mean: 432.79 tokens max: 512 tokens

• Samples:

  anchor	positive
  CHAPTER 39 Adrenocorticosteroids Adrenocortical Antagonists occurs. of /d of or in intermediate-, long-acting glucocorticoids greater growth-suppressing the steroid in larger than amounts, as cortisone hydrocortisone, which mineralocorticoid effects addition to glucocorticoid and fluid and loss of potassium. patients this a hypokalemic, and in blood pressure. hypoproteinemia, renal disease, liver disease, also occur. In patients with disease, small of may These by using non-salt-retaining and supplements. C. Suppression corticosteroids adrenal suppression occur. weeks the given appropriate at times dosage 24–48 hours) or stress ten-fold for or costeroid dosage be it slowly. If to reduction be slow levels. It take 2–12 to and cortisol may not to normal The suppression not treatment ACTH does time for normal function. the too receiving a certain disorder, the	CHAPTER 39 Adrenocorticosteroids & Adrenocortical Antagonists 707 hypertension also occurs. In dosages of 45 mg/m 2 /d or more of hydrocortisone or its equivalent, growth retardation occurs in children. Medium-, intermediate-, and long-acting glucocorticoids have greater growth-suppressing potency than the natural steroid at equivalent doses. When given in larger than physiologic amounts, steroids such as cortisone and hydrocortisone, which have mineralocorticoid effects in addition to glucocorticoid effects, cause some sodium and fluid retention and loss of potassium. In patients with normal cardiovas- cular and renal function, this leads to a hypokalemic, hypochloremic alkalosis and eventually to a rise in blood pressure. In patients with hypoproteinemia, renal disease, or liver disease, edema may also occur. In patients with heart disease, even small degrees of sodium retention may lead to heart failure. These effects can be minimized by using synthetic non-salt-retaining steroids, ...
  is a treatment not reduce the return function. dosage rapidly a certain the symptoms the in patients an disorder patients Cushing’s disease) symptoms with rapid symptoms include anorexia, vomit- ing, weight loss, postural reflect true glucocorticoid deficiency, occur in the normal or even plasma levels, sug- gesting glucocorticoids must carefully the hyperglycemia, sodium with edema hypertension, hypokalemia, peptic osteopo- rosis, and and intermittent alternate-day) can on this Even patients may of stress, surgical are or or acci- occur. B. with with peptic hypertension with failure, cer- as varicella tuberculosis, psycho- ses, osteoporosis, Glucocorticoid differ respect relative anti- inflammatory and mineralocorticoid of available ( Table and these factors should be in drug to used. ACTH Adrenocortical Steroids patients normal	is not a pituitary problem, and treatment with ACTH does not reduce the time required for the return of normal function. If the dosage is reduced too rapidly in patients receiving gluco- corticoids for a certain disorder, the symptoms of the disorder may reappear or increase in intensity. However, patients without an underlying disorder (eg, patients cured surgically of Cushing’s disease) also develop symptoms with rapid reductions in cortico- steroid levels. These symptoms include anorexia, nausea or vomit- ing, weight loss, lethargy, headache, fever, joint or muscle pain, and postural hypotension. Although many of these symptoms may reflect true glucocorticoid deficiency, they may also occur in the presence of normal or even elevated plasma cortisol levels, sug- gesting glucocorticoid dependence. Contraindications & Cautions A. Special Precautions Patients receiving glucocorticoids must be monitored carefully for the development of hyperglycemia, glycosuria, sodium retention with ede...
  ( Table and these should be taken in be A. ACTH ACTH used past production to However, when is able, ACTH therapeutic agent has abandoned. which claimed be effective than were due of of were dosage Dosage the regimen physician consider the disease, amount likely to required the effect, therapy. required for the dose to obtain initial the for needed effect be until a small or symptoms is When it is continuously plasma levels to ACTH, paren- preparation oral doses frequent The situation with respect use of inflammatory allergic The same total quantity few be effective many smaller slowly absorbed autoimmune involving organs aggressively, is as treatment. complexes macrophages, of predni- divided doses dosage is serious dosage can gradually large required prolonged time, after control When used manner, large amounts	available ( Table 39–1 ), and these factors should be taken into account in selecting the drug to be used. A. ACTH versus Adrenocortical Steroids In patients with normal adrenals, ACTH was used in the past to induce the endogenous production of cortisol to obtain similar effects. However, except when an increase in androgens is desir- able, the use of ACTH as a therapeutic agent has been abandoned. Instances in which ACTH was claimed to be more effective than glucocorticoids were probably due to the administration of smaller amounts of corticosteroids than were produced by the dosage of ACTH. B. Dosage In determining the dosage regimen to be used, the physician must consider the seriousness of the disease, the amount of drug likely to be required to obtain the desired effect, and the duration of therapy. In some diseases, the amount required for maintenance of the desired therapeutic effect is less than the dose needed to obtain the initial effect, and the lowest possible dosage for th...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• dataloader_drop_last: True
• dataloader_num_workers: 2
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: RikoteMaster/embedder-granite
• hub_strategy: end
• hub_private_repo: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 2
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: RikoteMaster/embedder-granite
• hub_strategy: end
• hub_private_repo: True
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.1859	50	0.3983	-
0.3717	100	0.193	-
0.5576	150	0.0828	-
0.7435	200	0.0409	0.0339
0.9294	250	0.0386	-
1.1152	300	0.0322	-
1.3011	350	0.0311	-
1.4870	400	0.0275	0.0167
1.6729	450	0.0252	-
1.8587	500	0.0254	-
2.0446	550	0.0254	-
2.2305	600	0.0227	0.0129
2.4164	650	0.0236	-
2.6022	700	0.0185	-
2.7881	750	0.0234	-
2.9740	800	0.0274	0.0118
3.1599	850	0.0208	-
3.3457	900	0.0245	-
3.5316	950	0.0242	-
3.7175	1000	0.0219	0.0112
3.9033	1050	0.0239	-
4.0892	1100	0.0223	-
4.2751	1150	0.0212	-
4.461	1200	0.0223	0.0107
4.6468	1250	0.0228	-
4.8327	1300	0.0196	-

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.17
• Sentence Transformers: 4.1.0
• Transformers: 4.52.3
• PyTorch: 2.7.0+cu126
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}