Uploaded lora adapters after finetuning on primevul

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ultival_mistral/checkpoint-4000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+ultival_mistral/checkpoint-5000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+ultival_mistral/checkpoint-6000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+ultival_mistral/checkpoint-6184/tokenizer.json filter=lfs diff=lfs merge=lfs -text

ultival_mistral/checkpoint-4000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Ministral-8B-Instruct-2410
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

ultival_mistral/checkpoint-4000/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Ministral-8B-Instruct-2410",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "gate_proj",
+    "up_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

ultival_mistral/checkpoint-4000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:698d9998ba9ac5614be57347cce32a6efd94fb9a4c4f85c88af56df8cc145037
+size 349243752

ultival_mistral/checkpoint-4000/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% if messages[1]['role'] == 'user' %}{{ '[INST] ' + messages[0]['content'] + ' ' + messages[1]['content'] + ' [/INST]' }}{% set loop_messages = messages[2:] %}{% else %}{{ '[INST] ' + messages[0]['content'] + ' [/INST]' }}{% set loop_messages = messages[1:] %}{% endif %}{% else %}{% set loop_messages = messages %}{% endif %}{% for message in loop_messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}

ultival_mistral/checkpoint-4000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3db69e6e07f78ea590887f34eb2ad205b28add7aa3d0f2cb7d1b79888dc7be83
+size 177919028

ultival_mistral/checkpoint-4000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd55c3b412ed63dda63d944e5148e12c72c1382a50017bf4d626548cc6ff19a6
+size 14244

ultival_mistral/checkpoint-4000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ad2700215ffb525ca35a0aac5f8a10bb15a4b447f330bbaba9667f852d4d6d9
+size 1064

ultival_mistral/checkpoint-4000/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

ultival_mistral/checkpoint-4000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be2965d3e435dbdef9376e5bdb3dcc2bfc7e271aa5152ae896cf53c7e920adf8
+size 17078235

ultival_mistral/checkpoint-4000/trainer_state.json

@@ -0,0 +1,1466 @@
+{
+  "best_global_step": 3000,
+  "best_metric": 0.5850913524627686,
+  "best_model_checkpoint": "./ultival_run2/checkpoint-3000",
+  "epoch": 1.2936610608020698,
+  "eval_steps": 1000,
+  "global_step": 4000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00646830530401035,
+      "grad_norm": 0.8976479172706604,
+      "learning_rate": 7.6e-06,
+      "loss": 1.5544,
+      "step": 20
+    },
+    {
+      "epoch": 0.0129366106080207,
+      "grad_norm": 0.726784348487854,
+      "learning_rate": 1.56e-05,
+      "loss": 1.242,
+      "step": 40
+    },
+    {
+      "epoch": 0.019404915912031046,
+      "grad_norm": 0.4917074143886566,
+      "learning_rate": 2.36e-05,
+      "loss": 0.6694,
+      "step": 60
+    },
+    {
+      "epoch": 0.0258732212160414,
+      "grad_norm": 0.3471176326274872,
+      "learning_rate": 3.16e-05,
+      "loss": 0.5334,
+      "step": 80
+    },
+    {
+      "epoch": 0.03234152652005175,
+      "grad_norm": 0.41759616136550903,
+      "learning_rate": 3.960000000000001e-05,
+      "loss": 0.5435,
+      "step": 100
+    },
+    {
+      "epoch": 0.03880983182406209,
+      "grad_norm": 0.3755040466785431,
+      "learning_rate": 4.76e-05,
+      "loss": 0.5684,
+      "step": 120
+    },
+    {
+      "epoch": 0.045278137128072445,
+      "grad_norm": 0.37058988213539124,
+      "learning_rate": 5.560000000000001e-05,
+      "loss": 0.5329,
+      "step": 140
+    },
+    {
+      "epoch": 0.0517464424320828,
+      "grad_norm": 0.42603132128715515,
+      "learning_rate": 6.36e-05,
+      "loss": 0.5457,
+      "step": 160
+    },
+    {
+      "epoch": 0.05821474773609314,
+      "grad_norm": 0.3154866099357605,
+      "learning_rate": 7.16e-05,
+      "loss": 0.5472,
+      "step": 180
+    },
+    {
+      "epoch": 0.0646830530401035,
+      "grad_norm": 0.3919835388660431,
+      "learning_rate": 7.960000000000001e-05,
+      "loss": 0.4675,
+      "step": 200
+    },
+    {
+      "epoch": 0.07115135834411385,
+      "grad_norm": 0.32180657982826233,
+      "learning_rate": 8.76e-05,
+      "loss": 0.5078,
+      "step": 220
+    },
+    {
+      "epoch": 0.07761966364812418,
+      "grad_norm": 0.3246982991695404,
+      "learning_rate": 9.56e-05,
+      "loss": 0.488,
+      "step": 240
+    },
+    {
+      "epoch": 0.08408796895213454,
+      "grad_norm": 0.341487318277359,
+      "learning_rate": 0.00010360000000000001,
+      "loss": 0.5058,
+      "step": 260
+    },
+    {
+      "epoch": 0.09055627425614489,
+      "grad_norm": 0.3597952127456665,
+      "learning_rate": 0.00011160000000000002,
+      "loss": 0.4574,
+      "step": 280
+    },
+    {
+      "epoch": 0.09702457956015524,
+      "grad_norm": 0.40912091732025146,
+      "learning_rate": 0.00011960000000000001,
+      "loss": 0.4931,
+      "step": 300
+    },
+    {
+      "epoch": 0.1034928848641656,
+      "grad_norm": 0.37430697679519653,
+      "learning_rate": 0.0001276,
+      "loss": 0.4724,
+      "step": 320
+    },
+    {
+      "epoch": 0.10996119016817593,
+      "grad_norm": 0.3920972943305969,
+      "learning_rate": 0.00013560000000000002,
+      "loss": 0.4997,
+      "step": 340
+    },
+    {
+      "epoch": 0.11642949547218628,
+      "grad_norm": 0.32511982321739197,
+      "learning_rate": 0.0001436,
+      "loss": 0.4889,
+      "step": 360
+    },
+    {
+      "epoch": 0.12289780077619664,
+      "grad_norm": 0.3926583528518677,
+      "learning_rate": 0.0001516,
+      "loss": 0.4772,
+      "step": 380
+    },
+    {
+      "epoch": 0.129366106080207,
+      "grad_norm": 0.3535909950733185,
+      "learning_rate": 0.0001596,
+      "loss": 0.465,
+      "step": 400
+    },
+    {
+      "epoch": 0.13583441138421734,
+      "grad_norm": 0.34381625056266785,
+      "learning_rate": 0.0001676,
+      "loss": 0.4934,
+      "step": 420
+    },
+    {
+      "epoch": 0.1423027166882277,
+      "grad_norm": 0.3753703236579895,
+      "learning_rate": 0.0001756,
+      "loss": 0.5068,
+      "step": 440
+    },
+    {
+      "epoch": 0.14877102199223805,
+      "grad_norm": 0.34895479679107666,
+      "learning_rate": 0.00018360000000000002,
+      "loss": 0.5186,
+      "step": 460
+    },
+    {
+      "epoch": 0.15523932729624837,
+      "grad_norm": 0.24720332026481628,
+      "learning_rate": 0.0001916,
+      "loss": 0.461,
+      "step": 480
+    },
+    {
+      "epoch": 0.16170763260025872,
+      "grad_norm": 0.3713696002960205,
+      "learning_rate": 0.0001996,
+      "loss": 0.4932,
+      "step": 500
+    },
+    {
+      "epoch": 0.16817593790426907,
+      "grad_norm": 0.3724994659423828,
+      "learning_rate": 0.00019933145672061927,
+      "loss": 0.5036,
+      "step": 520
+    },
+    {
+      "epoch": 0.17464424320827943,
+      "grad_norm": 0.3562956750392914,
+      "learning_rate": 0.00019862772695285012,
+      "loss": 0.4871,
+      "step": 540
+    },
+    {
+      "epoch": 0.18111254851228978,
+      "grad_norm": 0.331737756729126,
+      "learning_rate": 0.00019792399718508095,
+      "loss": 0.5178,
+      "step": 560
+    },
+    {
+      "epoch": 0.18758085381630013,
+      "grad_norm": 0.34049859642982483,
+      "learning_rate": 0.00019722026741731178,
+      "loss": 0.488,
+      "step": 580
+    },
+    {
+      "epoch": 0.19404915912031048,
+      "grad_norm": 0.36076661944389343,
+      "learning_rate": 0.00019651653764954258,
+      "loss": 0.4821,
+      "step": 600
+    },
+    {
+      "epoch": 0.20051746442432083,
+      "grad_norm": 0.3059399127960205,
+      "learning_rate": 0.0001958128078817734,
+      "loss": 0.4924,
+      "step": 620
+    },
+    {
+      "epoch": 0.2069857697283312,
+      "grad_norm": 0.30692920088768005,
+      "learning_rate": 0.00019510907811400423,
+      "loss": 0.4822,
+      "step": 640
+    },
+    {
+      "epoch": 0.21345407503234154,
+      "grad_norm": 0.3319448232650757,
+      "learning_rate": 0.00019440534834623506,
+      "loss": 0.4607,
+      "step": 660
+    },
+    {
+      "epoch": 0.21992238033635186,
+      "grad_norm": 0.3047769069671631,
+      "learning_rate": 0.00019370161857846588,
+      "loss": 0.5357,
+      "step": 680
+    },
+    {
+      "epoch": 0.22639068564036222,
+      "grad_norm": 0.4353258013725281,
+      "learning_rate": 0.00019299788881069668,
+      "loss": 0.4738,
+      "step": 700
+    },
+    {
+      "epoch": 0.23285899094437257,
+      "grad_norm": 0.30877336859703064,
+      "learning_rate": 0.00019229415904292754,
+      "loss": 0.5128,
+      "step": 720
+    },
+    {
+      "epoch": 0.23932729624838292,
+      "grad_norm": 0.3188530206680298,
+      "learning_rate": 0.00019159042927515836,
+      "loss": 0.4651,
+      "step": 740
+    },
+    {
+      "epoch": 0.24579560155239327,
+      "grad_norm": 0.40678778290748596,
+      "learning_rate": 0.0001908866995073892,
+      "loss": 0.5112,
+      "step": 760
+    },
+    {
+      "epoch": 0.2522639068564036,
+      "grad_norm": 0.34188327193260193,
+      "learning_rate": 0.00019018296973962,
+      "loss": 0.487,
+      "step": 780
+    },
+    {
+      "epoch": 0.258732212160414,
+      "grad_norm": 0.3002854585647583,
+      "learning_rate": 0.00018947923997185081,
+      "loss": 0.512,
+      "step": 800
+    },
+    {
+      "epoch": 0.2652005174644243,
+      "grad_norm": 0.35756272077560425,
+      "learning_rate": 0.00018877551020408164,
+      "loss": 0.4812,
+      "step": 820
+    },
+    {
+      "epoch": 0.2716688227684347,
+      "grad_norm": 0.33527258038520813,
+      "learning_rate": 0.00018807178043631247,
+      "loss": 0.4895,
+      "step": 840
+    },
+    {
+      "epoch": 0.278137128072445,
+      "grad_norm": 0.3472883105278015,
+      "learning_rate": 0.0001873680506685433,
+      "loss": 0.4671,
+      "step": 860
+    },
+    {
+      "epoch": 0.2846054333764554,
+      "grad_norm": 0.3604671061038971,
+      "learning_rate": 0.0001866643209007741,
+      "loss": 0.5233,
+      "step": 880
+    },
+    {
+      "epoch": 0.2910737386804657,
+      "grad_norm": 0.31021153926849365,
+      "learning_rate": 0.00018596059113300492,
+      "loss": 0.491,
+      "step": 900
+    },
+    {
+      "epoch": 0.2975420439844761,
+      "grad_norm": 0.6417067050933838,
+      "learning_rate": 0.00018525686136523577,
+      "loss": 0.4858,
+      "step": 920
+    },
+    {
+      "epoch": 0.3040103492884864,
+      "grad_norm": 0.33719781041145325,
+      "learning_rate": 0.0001845531315974666,
+      "loss": 0.4722,
+      "step": 940
+    },
+    {
+      "epoch": 0.31047865459249674,
+      "grad_norm": 0.3568169176578522,
+      "learning_rate": 0.0001838494018296974,
+      "loss": 0.5088,
+      "step": 960
+    },
+    {
+      "epoch": 0.3169469598965071,
+      "grad_norm": 0.42162615060806274,
+      "learning_rate": 0.00018314567206192823,
+      "loss": 0.5091,
+      "step": 980
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "grad_norm": 0.3212260603904724,
+      "learning_rate": 0.00018244194229415905,
+      "loss": 0.4654,
+      "step": 1000
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "eval_loss": 0.6167479753494263,
+      "eval_runtime": 1013.3659,
+      "eval_samples_per_second": 6.046,
+      "eval_steps_per_second": 0.756,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3298835705045278,
+      "grad_norm": 0.3184376060962677,
+      "learning_rate": 0.00018173821252638988,
+      "loss": 0.4712,
+      "step": 1020
+    },
+    {
+      "epoch": 0.33635187580853815,
+      "grad_norm": 0.3114645183086395,
+      "learning_rate": 0.0001810344827586207,
+      "loss": 0.4393,
+      "step": 1040
+    },
+    {
+      "epoch": 0.3428201811125485,
+      "grad_norm": 0.30122488737106323,
+      "learning_rate": 0.0001803307529908515,
+      "loss": 0.4882,
+      "step": 1060
+    },
+    {
+      "epoch": 0.34928848641655885,
+      "grad_norm": 0.2877393066883087,
+      "learning_rate": 0.00017962702322308233,
+      "loss": 0.413,
+      "step": 1080
+    },
+    {
+      "epoch": 0.35575679172056923,
+      "grad_norm": 0.33227887749671936,
+      "learning_rate": 0.00017892329345531319,
+      "loss": 0.4527,
+      "step": 1100
+    },
+    {
+      "epoch": 0.36222509702457956,
+      "grad_norm": 0.33423128724098206,
+      "learning_rate": 0.000178219563687544,
+      "loss": 0.4604,
+      "step": 1120
+    },
+    {
+      "epoch": 0.36869340232858994,
+      "grad_norm": 0.36922487616539,
+      "learning_rate": 0.0001775158339197748,
+      "loss": 0.5044,
+      "step": 1140
+    },
+    {
+      "epoch": 0.37516170763260026,
+      "grad_norm": 0.34816262125968933,
+      "learning_rate": 0.00017681210415200564,
+      "loss": 0.4898,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3816300129366106,
+      "grad_norm": 0.32086044549942017,
+      "learning_rate": 0.00017610837438423646,
+      "loss": 0.4647,
+      "step": 1180
+    },
+    {
+      "epoch": 0.38809831824062097,
+      "grad_norm": 0.3879236876964569,
+      "learning_rate": 0.0001754046446164673,
+      "loss": 0.5179,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3945666235446313,
+      "grad_norm": 0.2899722158908844,
+      "learning_rate": 0.00017470091484869812,
+      "loss": 0.4786,
+      "step": 1220
+    },
+    {
+      "epoch": 0.40103492884864167,
+      "grad_norm": 0.3844458758831024,
+      "learning_rate": 0.00017399718508092892,
+      "loss": 0.4513,
+      "step": 1240
+    },
+    {
+      "epoch": 0.407503234152652,
+      "grad_norm": 0.3678169548511505,
+      "learning_rate": 0.00017329345531315974,
+      "loss": 0.4789,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4139715394566624,
+      "grad_norm": 0.37138116359710693,
+      "learning_rate": 0.00017258972554539057,
+      "loss": 0.4694,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4204398447606727,
+      "grad_norm": 0.3604221045970917,
+      "learning_rate": 0.00017188599577762142,
+      "loss": 0.4496,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4269081500646831,
+      "grad_norm": 0.33961811661720276,
+      "learning_rate": 0.00017118226600985222,
+      "loss": 0.4696,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4333764553686934,
+      "grad_norm": 0.3244430124759674,
+      "learning_rate": 0.00017047853624208305,
+      "loss": 0.4322,
+      "step": 1340
+    },
+    {
+      "epoch": 0.4398447606727037,
+      "grad_norm": 0.2879267930984497,
+      "learning_rate": 0.00016977480647431388,
+      "loss": 0.4646,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4463130659767141,
+      "grad_norm": 0.3437345027923584,
+      "learning_rate": 0.0001690710767065447,
+      "loss": 0.4711,
+      "step": 1380
+    },
+    {
+      "epoch": 0.45278137128072443,
+      "grad_norm": 0.3166882395744324,
+      "learning_rate": 0.00016836734693877553,
+      "loss": 0.486,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4592496765847348,
+      "grad_norm": 0.3677555322647095,
+      "learning_rate": 0.00016766361717100633,
+      "loss": 0.4677,
+      "step": 1420
+    },
+    {
+      "epoch": 0.46571798188874514,
+      "grad_norm": 0.32657769322395325,
+      "learning_rate": 0.00016695988740323715,
+      "loss": 0.4649,
+      "step": 1440
+    },
+    {
+      "epoch": 0.4721862871927555,
+      "grad_norm": 0.2980731427669525,
+      "learning_rate": 0.00016625615763546798,
+      "loss": 0.4547,
+      "step": 1460
+    },
+    {
+      "epoch": 0.47865459249676584,
+      "grad_norm": 0.3690549433231354,
+      "learning_rate": 0.0001655524278676988,
+      "loss": 0.4512,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4851228978007762,
+      "grad_norm": 0.31456947326660156,
+      "learning_rate": 0.00016484869809992963,
+      "loss": 0.4675,
+      "step": 1500
+    },
+    {
+      "epoch": 0.49159120310478654,
+      "grad_norm": 0.3823802173137665,
+      "learning_rate": 0.00016414496833216046,
+      "loss": 0.4849,
+      "step": 1520
+    },
+    {
+      "epoch": 0.49805950840879687,
+      "grad_norm": 0.33280324935913086,
+      "learning_rate": 0.0001634412385643913,
+      "loss": 0.4876,
+      "step": 1540
+    },
+    {
+      "epoch": 0.5045278137128072,
+      "grad_norm": 0.28788062930107117,
+      "learning_rate": 0.00016273750879662211,
+      "loss": 0.426,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5109961190168176,
+      "grad_norm": 0.3809375762939453,
+      "learning_rate": 0.00016203377902885294,
+      "loss": 0.4676,
+      "step": 1580
+    },
+    {
+      "epoch": 0.517464424320828,
+      "grad_norm": 0.27940312027931213,
+      "learning_rate": 0.00016133004926108374,
+      "loss": 0.4413,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5239327296248383,
+      "grad_norm": 0.30946919322013855,
+      "learning_rate": 0.00016062631949331457,
+      "loss": 0.4395,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5304010349288486,
+      "grad_norm": 0.36931583285331726,
+      "learning_rate": 0.0001599225897255454,
+      "loss": 0.4565,
+      "step": 1640
+    },
+    {
+      "epoch": 0.536869340232859,
+      "grad_norm": 0.3652470111846924,
+      "learning_rate": 0.00015921885995777622,
+      "loss": 0.4522,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5433376455368694,
+      "grad_norm": 0.3060782551765442,
+      "learning_rate": 0.00015851513019000705,
+      "loss": 0.45,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5498059508408797,
+      "grad_norm": 0.3275803029537201,
+      "learning_rate": 0.00015781140042223787,
+      "loss": 0.445,
+      "step": 1700
+    },
+    {
+      "epoch": 0.55627425614489,
+      "grad_norm": 0.34865352511405945,
+      "learning_rate": 0.0001571076706544687,
+      "loss": 0.4306,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5627425614489003,
+      "grad_norm": 0.42525383830070496,
+      "learning_rate": 0.00015640394088669953,
+      "loss": 0.4544,
+      "step": 1740
+    },
+    {
+      "epoch": 0.5692108667529108,
+      "grad_norm": 0.32165050506591797,
+      "learning_rate": 0.00015570021111893035,
+      "loss": 0.484,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5756791720569211,
+      "grad_norm": 0.3213440477848053,
+      "learning_rate": 0.00015499648135116115,
+      "loss": 0.4469,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5821474773609314,
+      "grad_norm": 0.42835432291030884,
+      "learning_rate": 0.00015429275158339198,
+      "loss": 0.4592,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5886157826649417,
+      "grad_norm": 0.3861497938632965,
+      "learning_rate": 0.0001535890218156228,
+      "loss": 0.4607,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5950840879689522,
+      "grad_norm": 0.4156148135662079,
+      "learning_rate": 0.00015288529204785363,
+      "loss": 0.4541,
+      "step": 1840
+    },
+    {
+      "epoch": 0.6015523932729625,
+      "grad_norm": 0.32371824979782104,
+      "learning_rate": 0.00015218156228008446,
+      "loss": 0.4287,
+      "step": 1860
+    },
+    {
+      "epoch": 0.6080206985769728,
+      "grad_norm": 0.363788366317749,
+      "learning_rate": 0.00015147783251231528,
+      "loss": 0.428,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6144890038809832,
+      "grad_norm": 0.3932839334011078,
+      "learning_rate": 0.0001507741027445461,
+      "loss": 0.4359,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6209573091849935,
+      "grad_norm": 0.3009307086467743,
+      "learning_rate": 0.00015007037297677694,
+      "loss": 0.3976,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6274256144890039,
+      "grad_norm": 0.3484659194946289,
+      "learning_rate": 0.00014936664320900776,
+      "loss": 0.4584,
+      "step": 1940
+    },
+    {
+      "epoch": 0.6338939197930142,
+      "grad_norm": 0.33728504180908203,
+      "learning_rate": 0.00014866291344123856,
+      "loss": 0.4543,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6403622250970246,
+      "grad_norm": 0.3207024037837982,
+      "learning_rate": 0.0001479591836734694,
+      "loss": 0.4611,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "grad_norm": 0.29134076833724976,
+      "learning_rate": 0.00014725545390570022,
+      "loss": 0.439,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "eval_loss": 0.6003134250640869,
+      "eval_runtime": 968.0302,
+      "eval_samples_per_second": 6.329,
+      "eval_steps_per_second": 0.791,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6532988357050453,
+      "grad_norm": 0.3548874258995056,
+      "learning_rate": 0.00014655172413793104,
+      "loss": 0.4548,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6597671410090556,
+      "grad_norm": 0.3707274794578552,
+      "learning_rate": 0.00014584799437016187,
+      "loss": 0.4271,
+      "step": 2040
+    },
+    {
+      "epoch": 0.666235446313066,
+      "grad_norm": 0.37172695994377136,
+      "learning_rate": 0.0001451442646023927,
+      "loss": 0.4626,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6727037516170763,
+      "grad_norm": 0.3459303081035614,
+      "learning_rate": 0.00014444053483462352,
+      "loss": 0.4317,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6791720569210866,
+      "grad_norm": 0.3477213680744171,
+      "learning_rate": 0.00014373680506685435,
+      "loss": 0.4243,
+      "step": 2100
+    },
+    {
+      "epoch": 0.685640362225097,
+      "grad_norm": 0.31847599148750305,
+      "learning_rate": 0.00014303307529908517,
+      "loss": 0.4234,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6921086675291074,
+      "grad_norm": 0.3231470584869385,
+      "learning_rate": 0.00014232934553131597,
+      "loss": 0.4302,
+      "step": 2140
+    },
+    {
+      "epoch": 0.6985769728331177,
+      "grad_norm": 0.3745759427547455,
+      "learning_rate": 0.0001416256157635468,
+      "loss": 0.4533,
+      "step": 2160
+    },
+    {
+      "epoch": 0.705045278137128,
+      "grad_norm": 0.34834805130958557,
+      "learning_rate": 0.00014092188599577763,
+      "loss": 0.4748,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7115135834411385,
+      "grad_norm": 0.3675827085971832,
+      "learning_rate": 0.00014021815622800845,
+      "loss": 0.4401,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7179818887451488,
+      "grad_norm": 0.2839801013469696,
+      "learning_rate": 0.00013951442646023928,
+      "loss": 0.4332,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7244501940491591,
+      "grad_norm": 0.27037495374679565,
+      "learning_rate": 0.00013881069669247008,
+      "loss": 0.4382,
+      "step": 2240
+    },
+    {
+      "epoch": 0.7309184993531694,
+      "grad_norm": 0.34917208552360535,
+      "learning_rate": 0.00013810696692470093,
+      "loss": 0.4251,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7373868046571799,
+      "grad_norm": 0.39801281690597534,
+      "learning_rate": 0.00013740323715693176,
+      "loss": 0.4432,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7438551099611902,
+      "grad_norm": 0.8932256102561951,
+      "learning_rate": 0.00013669950738916259,
+      "loss": 0.4577,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7503234152652005,
+      "grad_norm": 0.35125109553337097,
+      "learning_rate": 0.00013599577762139339,
+      "loss": 0.4502,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7567917205692108,
+      "grad_norm": 0.3012046813964844,
+      "learning_rate": 0.0001352920478536242,
+      "loss": 0.4513,
+      "step": 2340
+    },
+    {
+      "epoch": 0.7632600258732212,
+      "grad_norm": 0.3242063820362091,
+      "learning_rate": 0.00013458831808585504,
+      "loss": 0.4559,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7697283311772316,
+      "grad_norm": 0.3523860275745392,
+      "learning_rate": 0.00013388458831808587,
+      "loss": 0.4434,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7761966364812419,
+      "grad_norm": 0.29160013794898987,
+      "learning_rate": 0.0001331808585503167,
+      "loss": 0.44,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7826649417852523,
+      "grad_norm": 0.3047971725463867,
+      "learning_rate": 0.0001324771287825475,
+      "loss": 0.4336,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7891332470892626,
+      "grad_norm": 0.3192126750946045,
+      "learning_rate": 0.00013177339901477832,
+      "loss": 0.4495,
+      "step": 2440
+    },
+    {
+      "epoch": 0.795601552393273,
+      "grad_norm": 0.37997299432754517,
+      "learning_rate": 0.00013106966924700917,
+      "loss": 0.4685,
+      "step": 2460
+    },
+    {
+      "epoch": 0.8020698576972833,
+      "grad_norm": 0.2811839282512665,
+      "learning_rate": 0.00013036593947924,
+      "loss": 0.4206,
+      "step": 2480
+    },
+    {
+      "epoch": 0.8085381630012937,
+      "grad_norm": 0.3121369779109955,
+      "learning_rate": 0.0001296622097114708,
+      "loss": 0.425,
+      "step": 2500
+    },
+    {
+      "epoch": 0.815006468305304,
+      "grad_norm": 0.3134746253490448,
+      "learning_rate": 0.00012895847994370162,
+      "loss": 0.4498,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8214747736093143,
+      "grad_norm": 0.36752375960350037,
+      "learning_rate": 0.00012825475017593245,
+      "loss": 0.4288,
+      "step": 2540
+    },
+    {
+      "epoch": 0.8279430789133247,
+      "grad_norm": 0.3915141224861145,
+      "learning_rate": 0.00012755102040816328,
+      "loss": 0.4605,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8344113842173351,
+      "grad_norm": 0.3662431538105011,
+      "learning_rate": 0.00012684729064039408,
+      "loss": 0.4194,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8408796895213454,
+      "grad_norm": 0.3387574851512909,
+      "learning_rate": 0.0001261435608726249,
+      "loss": 0.4258,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8473479948253557,
+      "grad_norm": 0.38025060296058655,
+      "learning_rate": 0.00012543983110485573,
+      "loss": 0.4291,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8538163001293662,
+      "grad_norm": 0.3517441749572754,
+      "learning_rate": 0.00012473610133708658,
+      "loss": 0.4384,
+      "step": 2640
+    },
+    {
+      "epoch": 0.8602846054333765,
+      "grad_norm": 0.33037152886390686,
+      "learning_rate": 0.0001240323715693174,
+      "loss": 0.4452,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8667529107373868,
+      "grad_norm": 0.3645654618740082,
+      "learning_rate": 0.0001233286418015482,
+      "loss": 0.3992,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8732212160413971,
+      "grad_norm": 0.32396814227104187,
+      "learning_rate": 0.00012262491203377904,
+      "loss": 0.4,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8796895213454075,
+      "grad_norm": 0.3377726078033447,
+      "learning_rate": 0.00012192118226600986,
+      "loss": 0.4306,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8861578266494179,
+      "grad_norm": 0.32345983386039734,
+      "learning_rate": 0.00012121745249824067,
+      "loss": 0.4225,
+      "step": 2740
+    },
+    {
+      "epoch": 0.8926261319534282,
+      "grad_norm": 0.34778884053230286,
+      "learning_rate": 0.0001205137227304715,
+      "loss": 0.4256,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8990944372574385,
+      "grad_norm": 0.3861066997051239,
+      "learning_rate": 0.00011980999296270233,
+      "loss": 0.4465,
+      "step": 2780
+    },
+    {
+      "epoch": 0.9055627425614489,
+      "grad_norm": 0.3882784843444824,
+      "learning_rate": 0.00011910626319493314,
+      "loss": 0.4026,
+      "step": 2800
+    },
+    {
+      "epoch": 0.9120310478654593,
+      "grad_norm": 0.4380407929420471,
+      "learning_rate": 0.00011840253342716397,
+      "loss": 0.4375,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9184993531694696,
+      "grad_norm": 0.35943394899368286,
+      "learning_rate": 0.00011769880365939481,
+      "loss": 0.4337,
+      "step": 2840
+    },
+    {
+      "epoch": 0.92496765847348,
+      "grad_norm": 0.3862978219985962,
+      "learning_rate": 0.00011699507389162563,
+      "loss": 0.3949,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9314359637774903,
+      "grad_norm": 0.34592369198799133,
+      "learning_rate": 0.00011629134412385645,
+      "loss": 0.3798,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9379042690815006,
+      "grad_norm": 0.34942877292633057,
+      "learning_rate": 0.00011558761435608727,
+      "loss": 0.4235,
+      "step": 2900
+    },
+    {
+      "epoch": 0.944372574385511,
+      "grad_norm": 0.28018903732299805,
+      "learning_rate": 0.00011488388458831809,
+      "loss": 0.4353,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9508408796895214,
+      "grad_norm": 0.3505672812461853,
+      "learning_rate": 0.00011418015482054891,
+      "loss": 0.3954,
+      "step": 2940
+    },
+    {
+      "epoch": 0.9573091849935317,
+      "grad_norm": 0.3264220654964447,
+      "learning_rate": 0.00011347642505277974,
+      "loss": 0.4422,
+      "step": 2960
+    },
+    {
+      "epoch": 0.963777490297542,
+      "grad_norm": 0.36382535099983215,
+      "learning_rate": 0.00011277269528501055,
+      "loss": 0.4116,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "grad_norm": 0.3051494359970093,
+      "learning_rate": 0.00011206896551724138,
+      "loss": 0.4135,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "eval_loss": 0.5850913524627686,
+      "eval_runtime": 967.6256,
+      "eval_samples_per_second": 6.332,
+      "eval_steps_per_second": 0.792,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9767141009055628,
+      "grad_norm": 0.34533271193504333,
+      "learning_rate": 0.00011136523574947222,
+      "loss": 0.4312,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9831824062095731,
+      "grad_norm": 0.35221609473228455,
+      "learning_rate": 0.00011066150598170305,
+      "loss": 0.4245,
+      "step": 3040
+    },
+    {
+      "epoch": 0.9896507115135834,
+      "grad_norm": 0.3893713057041168,
+      "learning_rate": 0.00010995777621393386,
+      "loss": 0.4048,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9961190168175937,
+      "grad_norm": 0.3586278557777405,
+      "learning_rate": 0.00010925404644616468,
+      "loss": 0.4159,
+      "step": 3080
+    },
+    {
+      "epoch": 1.0025873221216042,
+      "grad_norm": 0.32620444893836975,
+      "learning_rate": 0.0001085503166783955,
+      "loss": 0.3741,
+      "step": 3100
+    },
+    {
+      "epoch": 1.0090556274256144,
+      "grad_norm": 0.33812645077705383,
+      "learning_rate": 0.00010784658691062632,
+      "loss": 0.3694,
+      "step": 3120
+    },
+    {
+      "epoch": 1.0155239327296248,
+      "grad_norm": 0.39038708806037903,
+      "learning_rate": 0.00010714285714285715,
+      "loss": 0.3882,
+      "step": 3140
+    },
+    {
+      "epoch": 1.0219922380336353,
+      "grad_norm": 0.3209969997406006,
+      "learning_rate": 0.00010643912737508796,
+      "loss": 0.3437,
+      "step": 3160
+    },
+    {
+      "epoch": 1.0284605433376455,
+      "grad_norm": 0.3705078661441803,
+      "learning_rate": 0.00010573539760731879,
+      "loss": 0.3568,
+      "step": 3180
+    },
+    {
+      "epoch": 1.034928848641656,
+      "grad_norm": 0.3562203645706177,
+      "learning_rate": 0.0001050316678395496,
+      "loss": 0.3916,
+      "step": 3200
+    },
+    {
+      "epoch": 1.0413971539456663,
+      "grad_norm": 0.3724074363708496,
+      "learning_rate": 0.00010432793807178046,
+      "loss": 0.3866,
+      "step": 3220
+    },
+    {
+      "epoch": 1.0478654592496766,
+      "grad_norm": 0.40453559160232544,
+      "learning_rate": 0.00010362420830401127,
+      "loss": 0.368,
+      "step": 3240
+    },
+    {
+      "epoch": 1.054333764553687,
+      "grad_norm": 0.3316144049167633,
+      "learning_rate": 0.0001029204785362421,
+      "loss": 0.3696,
+      "step": 3260
+    },
+    {
+      "epoch": 1.0608020698576972,
+      "grad_norm": 0.3503650724887848,
+      "learning_rate": 0.00010221674876847291,
+      "loss": 0.3644,
+      "step": 3280
+    },
+    {
+      "epoch": 1.0672703751617076,
+      "grad_norm": 0.3136923909187317,
+      "learning_rate": 0.00010151301900070374,
+      "loss": 0.3414,
+      "step": 3300
+    },
+    {
+      "epoch": 1.073738680465718,
+      "grad_norm": 0.3573864996433258,
+      "learning_rate": 0.00010080928923293456,
+      "loss": 0.3586,
+      "step": 3320
+    },
+    {
+      "epoch": 1.0802069857697283,
+      "grad_norm": 0.26215964555740356,
+      "learning_rate": 0.00010010555946516538,
+      "loss": 0.356,
+      "step": 3340
+    },
+    {
+      "epoch": 1.0866752910737387,
+      "grad_norm": 0.3103466331958771,
+      "learning_rate": 9.940182969739622e-05,
+      "loss": 0.3478,
+      "step": 3360
+    },
+    {
+      "epoch": 1.093143596377749,
+      "grad_norm": 0.3601842224597931,
+      "learning_rate": 9.869809992962703e-05,
+      "loss": 0.3635,
+      "step": 3380
+    },
+    {
+      "epoch": 1.0996119016817594,
+      "grad_norm": 0.3715918958187103,
+      "learning_rate": 9.799437016185785e-05,
+      "loss": 0.3489,
+      "step": 3400
+    },
+    {
+      "epoch": 1.1060802069857698,
+      "grad_norm": 0.4216213524341583,
+      "learning_rate": 9.729064039408867e-05,
+      "loss": 0.3461,
+      "step": 3420
+    },
+    {
+      "epoch": 1.11254851228978,
+      "grad_norm": 0.3878704905509949,
+      "learning_rate": 9.65869106263195e-05,
+      "loss": 0.3548,
+      "step": 3440
+    },
+    {
+      "epoch": 1.1190168175937905,
+      "grad_norm": 0.36658766865730286,
+      "learning_rate": 9.588318085855032e-05,
+      "loss": 0.3395,
+      "step": 3460
+    },
+    {
+      "epoch": 1.1254851228978007,
+      "grad_norm": 0.38134855031967163,
+      "learning_rate": 9.517945109078115e-05,
+      "loss": 0.3829,
+      "step": 3480
+    },
+    {
+      "epoch": 1.131953428201811,
+      "grad_norm": 0.42683905363082886,
+      "learning_rate": 9.447572132301197e-05,
+      "loss": 0.3914,
+      "step": 3500
+    },
+    {
+      "epoch": 1.1384217335058215,
+      "grad_norm": 0.3340086042881012,
+      "learning_rate": 9.377199155524279e-05,
+      "loss": 0.3301,
+      "step": 3520
+    },
+    {
+      "epoch": 1.1448900388098318,
+      "grad_norm": 0.37897104024887085,
+      "learning_rate": 9.306826178747361e-05,
+      "loss": 0.3536,
+      "step": 3540
+    },
+    {
+      "epoch": 1.1513583441138422,
+      "grad_norm": 0.361061692237854,
+      "learning_rate": 9.236453201970444e-05,
+      "loss": 0.338,
+      "step": 3560
+    },
+    {
+      "epoch": 1.1578266494178524,
+      "grad_norm": 0.34730997681617737,
+      "learning_rate": 9.166080225193527e-05,
+      "loss": 0.3679,
+      "step": 3580
+    },
+    {
+      "epoch": 1.1642949547218628,
+      "grad_norm": 0.29455122351646423,
+      "learning_rate": 9.095707248416608e-05,
+      "loss": 0.3602,
+      "step": 3600
+    },
+    {
+      "epoch": 1.1707632600258733,
+      "grad_norm": 0.33159056305885315,
+      "learning_rate": 9.02533427163969e-05,
+      "loss": 0.3359,
+      "step": 3620
+    },
+    {
+      "epoch": 1.1772315653298835,
+      "grad_norm": 0.48726096749305725,
+      "learning_rate": 8.954961294862773e-05,
+      "loss": 0.3879,
+      "step": 3640
+    },
+    {
+      "epoch": 1.183699870633894,
+      "grad_norm": 0.3182481825351715,
+      "learning_rate": 8.884588318085856e-05,
+      "loss": 0.3392,
+      "step": 3660
+    },
+    {
+      "epoch": 1.1901681759379044,
+      "grad_norm": 0.3488766551017761,
+      "learning_rate": 8.814215341308939e-05,
+      "loss": 0.3419,
+      "step": 3680
+    },
+    {
+      "epoch": 1.1966364812419146,
+      "grad_norm": 0.34345540404319763,
+      "learning_rate": 8.74384236453202e-05,
+      "loss": 0.3598,
+      "step": 3700
+    },
+    {
+      "epoch": 1.203104786545925,
+      "grad_norm": 0.37642374634742737,
+      "learning_rate": 8.673469387755102e-05,
+      "loss": 0.3578,
+      "step": 3720
+    },
+    {
+      "epoch": 1.2095730918499352,
+      "grad_norm": 0.3726005554199219,
+      "learning_rate": 8.603096410978185e-05,
+      "loss": 0.3547,
+      "step": 3740
+    },
+    {
+      "epoch": 1.2160413971539457,
+      "grad_norm": 0.36603832244873047,
+      "learning_rate": 8.532723434201268e-05,
+      "loss": 0.3831,
+      "step": 3760
+    },
+    {
+      "epoch": 1.222509702457956,
+      "grad_norm": 0.345103919506073,
+      "learning_rate": 8.462350457424349e-05,
+      "loss": 0.3563,
+      "step": 3780
+    },
+    {
+      "epoch": 1.2289780077619663,
+      "grad_norm": 0.31071504950523376,
+      "learning_rate": 8.391977480647432e-05,
+      "loss": 0.3263,
+      "step": 3800
+    },
+    {
+      "epoch": 1.2354463130659767,
+      "grad_norm": 0.3534263074398041,
+      "learning_rate": 8.321604503870513e-05,
+      "loss": 0.3591,
+      "step": 3820
+    },
+    {
+      "epoch": 1.2419146183699872,
+      "grad_norm": 0.3568720817565918,
+      "learning_rate": 8.251231527093597e-05,
+      "loss": 0.3815,
+      "step": 3840
+    },
+    {
+      "epoch": 1.2483829236739974,
+      "grad_norm": 0.3450963795185089,
+      "learning_rate": 8.18085855031668e-05,
+      "loss": 0.3481,
+      "step": 3860
+    },
+    {
+      "epoch": 1.2548512289780078,
+      "grad_norm": 0.3554648160934448,
+      "learning_rate": 8.110485573539761e-05,
+      "loss": 0.3364,
+      "step": 3880
+    },
+    {
+      "epoch": 1.261319534282018,
+      "grad_norm": 0.4188801348209381,
+      "learning_rate": 8.040112596762844e-05,
+      "loss": 0.3686,
+      "step": 3900
+    },
+    {
+      "epoch": 1.2677878395860285,
+      "grad_norm": 0.3129253089427948,
+      "learning_rate": 7.969739619985925e-05,
+      "loss": 0.3664,
+      "step": 3920
+    },
+    {
+      "epoch": 1.274256144890039,
+      "grad_norm": 0.3885388970375061,
+      "learning_rate": 7.899366643209009e-05,
+      "loss": 0.3415,
+      "step": 3940
+    },
+    {
+      "epoch": 1.2807244501940491,
+      "grad_norm": 0.4131453335285187,
+      "learning_rate": 7.82899366643209e-05,
+      "loss": 0.3552,
+      "step": 3960
+    },
+    {
+      "epoch": 1.2871927554980596,
+      "grad_norm": 0.39486396312713623,
+      "learning_rate": 7.758620689655173e-05,
+      "loss": 0.366,
+      "step": 3980
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "grad_norm": 0.3580794930458069,
+      "learning_rate": 7.688247712878254e-05,
+      "loss": 0.3585,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "eval_loss": 0.5952740907669067,
+      "eval_runtime": 967.8131,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 4000
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.33128270139818e+18,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

ultival_mistral/checkpoint-4000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb8e5db938f70ff2ddfe787ade649d14bb55cc2be7a65bd63a20c3303bf68492
+size 5368

ultival_mistral/checkpoint-5000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Ministral-8B-Instruct-2410
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

ultival_mistral/checkpoint-5000/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Ministral-8B-Instruct-2410",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "gate_proj",
+    "up_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

ultival_mistral/checkpoint-5000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d4e5d637b24ecf43d167a40f0933906906cec8c26ae36a745ac7a45e03349e3
+size 349243752

ultival_mistral/checkpoint-5000/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% if messages[1]['role'] == 'user' %}{{ '[INST] ' + messages[0]['content'] + ' ' + messages[1]['content'] + ' [/INST]' }}{% set loop_messages = messages[2:] %}{% else %}{{ '[INST] ' + messages[0]['content'] + ' [/INST]' }}{% set loop_messages = messages[1:] %}{% endif %}{% else %}{% set loop_messages = messages %}{% endif %}{% for message in loop_messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}

ultival_mistral/checkpoint-5000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69e7979fd79b7efdd90afb4a0ff92abf491ca11ea63da3fa64e3f46b1d4b47b0
+size 177919028

ultival_mistral/checkpoint-5000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:301bae54f3043b54f78ee582aa05b8bf01996d027ad704f9a95f10d8be516262
+size 14244

ultival_mistral/checkpoint-5000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90de424fe588568c8644c48a645a04810ccb62dd370abc6f52d176fcc605dfa4
+size 1064

ultival_mistral/checkpoint-5000/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

ultival_mistral/checkpoint-5000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be2965d3e435dbdef9376e5bdb3dcc2bfc7e271aa5152ae896cf53c7e920adf8
+size 17078235

ultival_mistral/checkpoint-5000/trainer_state.json

@@ -0,0 +1,1824 @@
+{
+  "best_global_step": 3000,
+  "best_metric": 0.5850913524627686,
+  "best_model_checkpoint": "./ultival_run2/checkpoint-3000",
+  "epoch": 1.6170763260025873,
+  "eval_steps": 1000,
+  "global_step": 5000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00646830530401035,
+      "grad_norm": 0.8976479172706604,
+      "learning_rate": 7.6e-06,
+      "loss": 1.5544,
+      "step": 20
+    },
+    {
+      "epoch": 0.0129366106080207,
+      "grad_norm": 0.726784348487854,
+      "learning_rate": 1.56e-05,
+      "loss": 1.242,
+      "step": 40
+    },
+    {
+      "epoch": 0.019404915912031046,
+      "grad_norm": 0.4917074143886566,
+      "learning_rate": 2.36e-05,
+      "loss": 0.6694,
+      "step": 60
+    },
+    {
+      "epoch": 0.0258732212160414,
+      "grad_norm": 0.3471176326274872,
+      "learning_rate": 3.16e-05,
+      "loss": 0.5334,
+      "step": 80
+    },
+    {
+      "epoch": 0.03234152652005175,
+      "grad_norm": 0.41759616136550903,
+      "learning_rate": 3.960000000000001e-05,
+      "loss": 0.5435,
+      "step": 100
+    },
+    {
+      "epoch": 0.03880983182406209,
+      "grad_norm": 0.3755040466785431,
+      "learning_rate": 4.76e-05,
+      "loss": 0.5684,
+      "step": 120
+    },
+    {
+      "epoch": 0.045278137128072445,
+      "grad_norm": 0.37058988213539124,
+      "learning_rate": 5.560000000000001e-05,
+      "loss": 0.5329,
+      "step": 140
+    },
+    {
+      "epoch": 0.0517464424320828,
+      "grad_norm": 0.42603132128715515,
+      "learning_rate": 6.36e-05,
+      "loss": 0.5457,
+      "step": 160
+    },
+    {
+      "epoch": 0.05821474773609314,
+      "grad_norm": 0.3154866099357605,
+      "learning_rate": 7.16e-05,
+      "loss": 0.5472,
+      "step": 180
+    },
+    {
+      "epoch": 0.0646830530401035,
+      "grad_norm": 0.3919835388660431,
+      "learning_rate": 7.960000000000001e-05,
+      "loss": 0.4675,
+      "step": 200
+    },
+    {
+      "epoch": 0.07115135834411385,
+      "grad_norm": 0.32180657982826233,
+      "learning_rate": 8.76e-05,
+      "loss": 0.5078,
+      "step": 220
+    },
+    {
+      "epoch": 0.07761966364812418,
+      "grad_norm": 0.3246982991695404,
+      "learning_rate": 9.56e-05,
+      "loss": 0.488,
+      "step": 240
+    },
+    {
+      "epoch": 0.08408796895213454,
+      "grad_norm": 0.341487318277359,
+      "learning_rate": 0.00010360000000000001,
+      "loss": 0.5058,
+      "step": 260
+    },
+    {
+      "epoch": 0.09055627425614489,
+      "grad_norm": 0.3597952127456665,
+      "learning_rate": 0.00011160000000000002,
+      "loss": 0.4574,
+      "step": 280
+    },
+    {
+      "epoch": 0.09702457956015524,
+      "grad_norm": 0.40912091732025146,
+      "learning_rate": 0.00011960000000000001,
+      "loss": 0.4931,
+      "step": 300
+    },
+    {
+      "epoch": 0.1034928848641656,
+      "grad_norm": 0.37430697679519653,
+      "learning_rate": 0.0001276,
+      "loss": 0.4724,
+      "step": 320
+    },
+    {
+      "epoch": 0.10996119016817593,
+      "grad_norm": 0.3920972943305969,
+      "learning_rate": 0.00013560000000000002,
+      "loss": 0.4997,
+      "step": 340
+    },
+    {
+      "epoch": 0.11642949547218628,
+      "grad_norm": 0.32511982321739197,
+      "learning_rate": 0.0001436,
+      "loss": 0.4889,
+      "step": 360
+    },
+    {
+      "epoch": 0.12289780077619664,
+      "grad_norm": 0.3926583528518677,
+      "learning_rate": 0.0001516,
+      "loss": 0.4772,
+      "step": 380
+    },
+    {
+      "epoch": 0.129366106080207,
+      "grad_norm": 0.3535909950733185,
+      "learning_rate": 0.0001596,
+      "loss": 0.465,
+      "step": 400
+    },
+    {
+      "epoch": 0.13583441138421734,
+      "grad_norm": 0.34381625056266785,
+      "learning_rate": 0.0001676,
+      "loss": 0.4934,
+      "step": 420
+    },
+    {
+      "epoch": 0.1423027166882277,
+      "grad_norm": 0.3753703236579895,
+      "learning_rate": 0.0001756,
+      "loss": 0.5068,
+      "step": 440
+    },
+    {
+      "epoch": 0.14877102199223805,
+      "grad_norm": 0.34895479679107666,
+      "learning_rate": 0.00018360000000000002,
+      "loss": 0.5186,
+      "step": 460
+    },
+    {
+      "epoch": 0.15523932729624837,
+      "grad_norm": 0.24720332026481628,
+      "learning_rate": 0.0001916,
+      "loss": 0.461,
+      "step": 480
+    },
+    {
+      "epoch": 0.16170763260025872,
+      "grad_norm": 0.3713696002960205,
+      "learning_rate": 0.0001996,
+      "loss": 0.4932,
+      "step": 500
+    },
+    {
+      "epoch": 0.16817593790426907,
+      "grad_norm": 0.3724994659423828,
+      "learning_rate": 0.00019933145672061927,
+      "loss": 0.5036,
+      "step": 520
+    },
+    {
+      "epoch": 0.17464424320827943,
+      "grad_norm": 0.3562956750392914,
+      "learning_rate": 0.00019862772695285012,
+      "loss": 0.4871,
+      "step": 540
+    },
+    {
+      "epoch": 0.18111254851228978,
+      "grad_norm": 0.331737756729126,
+      "learning_rate": 0.00019792399718508095,
+      "loss": 0.5178,
+      "step": 560
+    },
+    {
+      "epoch": 0.18758085381630013,
+      "grad_norm": 0.34049859642982483,
+      "learning_rate": 0.00019722026741731178,
+      "loss": 0.488,
+      "step": 580
+    },
+    {
+      "epoch": 0.19404915912031048,
+      "grad_norm": 0.36076661944389343,
+      "learning_rate": 0.00019651653764954258,
+      "loss": 0.4821,
+      "step": 600
+    },
+    {
+      "epoch": 0.20051746442432083,
+      "grad_norm": 0.3059399127960205,
+      "learning_rate": 0.0001958128078817734,
+      "loss": 0.4924,
+      "step": 620
+    },
+    {
+      "epoch": 0.2069857697283312,
+      "grad_norm": 0.30692920088768005,
+      "learning_rate": 0.00019510907811400423,
+      "loss": 0.4822,
+      "step": 640
+    },
+    {
+      "epoch": 0.21345407503234154,
+      "grad_norm": 0.3319448232650757,
+      "learning_rate": 0.00019440534834623506,
+      "loss": 0.4607,
+      "step": 660
+    },
+    {
+      "epoch": 0.21992238033635186,
+      "grad_norm": 0.3047769069671631,
+      "learning_rate": 0.00019370161857846588,
+      "loss": 0.5357,
+      "step": 680
+    },
+    {
+      "epoch": 0.22639068564036222,
+      "grad_norm": 0.4353258013725281,
+      "learning_rate": 0.00019299788881069668,
+      "loss": 0.4738,
+      "step": 700
+    },
+    {
+      "epoch": 0.23285899094437257,
+      "grad_norm": 0.30877336859703064,
+      "learning_rate": 0.00019229415904292754,
+      "loss": 0.5128,
+      "step": 720
+    },
+    {
+      "epoch": 0.23932729624838292,
+      "grad_norm": 0.3188530206680298,
+      "learning_rate": 0.00019159042927515836,
+      "loss": 0.4651,
+      "step": 740
+    },
+    {
+      "epoch": 0.24579560155239327,
+      "grad_norm": 0.40678778290748596,
+      "learning_rate": 0.0001908866995073892,
+      "loss": 0.5112,
+      "step": 760
+    },
+    {
+      "epoch": 0.2522639068564036,
+      "grad_norm": 0.34188327193260193,
+      "learning_rate": 0.00019018296973962,
+      "loss": 0.487,
+      "step": 780
+    },
+    {
+      "epoch": 0.258732212160414,
+      "grad_norm": 0.3002854585647583,
+      "learning_rate": 0.00018947923997185081,
+      "loss": 0.512,
+      "step": 800
+    },
+    {
+      "epoch": 0.2652005174644243,
+      "grad_norm": 0.35756272077560425,
+      "learning_rate": 0.00018877551020408164,
+      "loss": 0.4812,
+      "step": 820
+    },
+    {
+      "epoch": 0.2716688227684347,
+      "grad_norm": 0.33527258038520813,
+      "learning_rate": 0.00018807178043631247,
+      "loss": 0.4895,
+      "step": 840
+    },
+    {
+      "epoch": 0.278137128072445,
+      "grad_norm": 0.3472883105278015,
+      "learning_rate": 0.0001873680506685433,
+      "loss": 0.4671,
+      "step": 860
+    },
+    {
+      "epoch": 0.2846054333764554,
+      "grad_norm": 0.3604671061038971,
+      "learning_rate": 0.0001866643209007741,
+      "loss": 0.5233,
+      "step": 880
+    },
+    {
+      "epoch": 0.2910737386804657,
+      "grad_norm": 0.31021153926849365,
+      "learning_rate": 0.00018596059113300492,
+      "loss": 0.491,
+      "step": 900
+    },
+    {
+      "epoch": 0.2975420439844761,
+      "grad_norm": 0.6417067050933838,
+      "learning_rate": 0.00018525686136523577,
+      "loss": 0.4858,
+      "step": 920
+    },
+    {
+      "epoch": 0.3040103492884864,
+      "grad_norm": 0.33719781041145325,
+      "learning_rate": 0.0001845531315974666,
+      "loss": 0.4722,
+      "step": 940
+    },
+    {
+      "epoch": 0.31047865459249674,
+      "grad_norm": 0.3568169176578522,
+      "learning_rate": 0.0001838494018296974,
+      "loss": 0.5088,
+      "step": 960
+    },
+    {
+      "epoch": 0.3169469598965071,
+      "grad_norm": 0.42162615060806274,
+      "learning_rate": 0.00018314567206192823,
+      "loss": 0.5091,
+      "step": 980
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "grad_norm": 0.3212260603904724,
+      "learning_rate": 0.00018244194229415905,
+      "loss": 0.4654,
+      "step": 1000
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "eval_loss": 0.6167479753494263,
+      "eval_runtime": 1013.3659,
+      "eval_samples_per_second": 6.046,
+      "eval_steps_per_second": 0.756,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3298835705045278,
+      "grad_norm": 0.3184376060962677,
+      "learning_rate": 0.00018173821252638988,
+      "loss": 0.4712,
+      "step": 1020
+    },
+    {
+      "epoch": 0.33635187580853815,
+      "grad_norm": 0.3114645183086395,
+      "learning_rate": 0.0001810344827586207,
+      "loss": 0.4393,
+      "step": 1040
+    },
+    {
+      "epoch": 0.3428201811125485,
+      "grad_norm": 0.30122488737106323,
+      "learning_rate": 0.0001803307529908515,
+      "loss": 0.4882,
+      "step": 1060
+    },
+    {
+      "epoch": 0.34928848641655885,
+      "grad_norm": 0.2877393066883087,
+      "learning_rate": 0.00017962702322308233,
+      "loss": 0.413,
+      "step": 1080
+    },
+    {
+      "epoch": 0.35575679172056923,
+      "grad_norm": 0.33227887749671936,
+      "learning_rate": 0.00017892329345531319,
+      "loss": 0.4527,
+      "step": 1100
+    },
+    {
+      "epoch": 0.36222509702457956,
+      "grad_norm": 0.33423128724098206,
+      "learning_rate": 0.000178219563687544,
+      "loss": 0.4604,
+      "step": 1120
+    },
+    {
+      "epoch": 0.36869340232858994,
+      "grad_norm": 0.36922487616539,
+      "learning_rate": 0.0001775158339197748,
+      "loss": 0.5044,
+      "step": 1140
+    },
+    {
+      "epoch": 0.37516170763260026,
+      "grad_norm": 0.34816262125968933,
+      "learning_rate": 0.00017681210415200564,
+      "loss": 0.4898,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3816300129366106,
+      "grad_norm": 0.32086044549942017,
+      "learning_rate": 0.00017610837438423646,
+      "loss": 0.4647,
+      "step": 1180
+    },
+    {
+      "epoch": 0.38809831824062097,
+      "grad_norm": 0.3879236876964569,
+      "learning_rate": 0.0001754046446164673,
+      "loss": 0.5179,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3945666235446313,
+      "grad_norm": 0.2899722158908844,
+      "learning_rate": 0.00017470091484869812,
+      "loss": 0.4786,
+      "step": 1220
+    },
+    {
+      "epoch": 0.40103492884864167,
+      "grad_norm": 0.3844458758831024,
+      "learning_rate": 0.00017399718508092892,
+      "loss": 0.4513,
+      "step": 1240
+    },
+    {
+      "epoch": 0.407503234152652,
+      "grad_norm": 0.3678169548511505,
+      "learning_rate": 0.00017329345531315974,
+      "loss": 0.4789,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4139715394566624,
+      "grad_norm": 0.37138116359710693,
+      "learning_rate": 0.00017258972554539057,
+      "loss": 0.4694,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4204398447606727,
+      "grad_norm": 0.3604221045970917,
+      "learning_rate": 0.00017188599577762142,
+      "loss": 0.4496,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4269081500646831,
+      "grad_norm": 0.33961811661720276,
+      "learning_rate": 0.00017118226600985222,
+      "loss": 0.4696,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4333764553686934,
+      "grad_norm": 0.3244430124759674,
+      "learning_rate": 0.00017047853624208305,
+      "loss": 0.4322,
+      "step": 1340
+    },
+    {
+      "epoch": 0.4398447606727037,
+      "grad_norm": 0.2879267930984497,
+      "learning_rate": 0.00016977480647431388,
+      "loss": 0.4646,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4463130659767141,
+      "grad_norm": 0.3437345027923584,
+      "learning_rate": 0.0001690710767065447,
+      "loss": 0.4711,
+      "step": 1380
+    },
+    {
+      "epoch": 0.45278137128072443,
+      "grad_norm": 0.3166882395744324,
+      "learning_rate": 0.00016836734693877553,
+      "loss": 0.486,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4592496765847348,
+      "grad_norm": 0.3677555322647095,
+      "learning_rate": 0.00016766361717100633,
+      "loss": 0.4677,
+      "step": 1420
+    },
+    {
+      "epoch": 0.46571798188874514,
+      "grad_norm": 0.32657769322395325,
+      "learning_rate": 0.00016695988740323715,
+      "loss": 0.4649,
+      "step": 1440
+    },
+    {
+      "epoch": 0.4721862871927555,
+      "grad_norm": 0.2980731427669525,
+      "learning_rate": 0.00016625615763546798,
+      "loss": 0.4547,
+      "step": 1460
+    },
+    {
+      "epoch": 0.47865459249676584,
+      "grad_norm": 0.3690549433231354,
+      "learning_rate": 0.0001655524278676988,
+      "loss": 0.4512,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4851228978007762,
+      "grad_norm": 0.31456947326660156,
+      "learning_rate": 0.00016484869809992963,
+      "loss": 0.4675,
+      "step": 1500
+    },
+    {
+      "epoch": 0.49159120310478654,
+      "grad_norm": 0.3823802173137665,
+      "learning_rate": 0.00016414496833216046,
+      "loss": 0.4849,
+      "step": 1520
+    },
+    {
+      "epoch": 0.49805950840879687,
+      "grad_norm": 0.33280324935913086,
+      "learning_rate": 0.0001634412385643913,
+      "loss": 0.4876,
+      "step": 1540
+    },
+    {
+      "epoch": 0.5045278137128072,
+      "grad_norm": 0.28788062930107117,
+      "learning_rate": 0.00016273750879662211,
+      "loss": 0.426,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5109961190168176,
+      "grad_norm": 0.3809375762939453,
+      "learning_rate": 0.00016203377902885294,
+      "loss": 0.4676,
+      "step": 1580
+    },
+    {
+      "epoch": 0.517464424320828,
+      "grad_norm": 0.27940312027931213,
+      "learning_rate": 0.00016133004926108374,
+      "loss": 0.4413,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5239327296248383,
+      "grad_norm": 0.30946919322013855,
+      "learning_rate": 0.00016062631949331457,
+      "loss": 0.4395,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5304010349288486,
+      "grad_norm": 0.36931583285331726,
+      "learning_rate": 0.0001599225897255454,
+      "loss": 0.4565,
+      "step": 1640
+    },
+    {
+      "epoch": 0.536869340232859,
+      "grad_norm": 0.3652470111846924,
+      "learning_rate": 0.00015921885995777622,
+      "loss": 0.4522,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5433376455368694,
+      "grad_norm": 0.3060782551765442,
+      "learning_rate": 0.00015851513019000705,
+      "loss": 0.45,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5498059508408797,
+      "grad_norm": 0.3275803029537201,
+      "learning_rate": 0.00015781140042223787,
+      "loss": 0.445,
+      "step": 1700
+    },
+    {
+      "epoch": 0.55627425614489,
+      "grad_norm": 0.34865352511405945,
+      "learning_rate": 0.0001571076706544687,
+      "loss": 0.4306,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5627425614489003,
+      "grad_norm": 0.42525383830070496,
+      "learning_rate": 0.00015640394088669953,
+      "loss": 0.4544,
+      "step": 1740
+    },
+    {
+      "epoch": 0.5692108667529108,
+      "grad_norm": 0.32165050506591797,
+      "learning_rate": 0.00015570021111893035,
+      "loss": 0.484,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5756791720569211,
+      "grad_norm": 0.3213440477848053,
+      "learning_rate": 0.00015499648135116115,
+      "loss": 0.4469,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5821474773609314,
+      "grad_norm": 0.42835432291030884,
+      "learning_rate": 0.00015429275158339198,
+      "loss": 0.4592,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5886157826649417,
+      "grad_norm": 0.3861497938632965,
+      "learning_rate": 0.0001535890218156228,
+      "loss": 0.4607,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5950840879689522,
+      "grad_norm": 0.4156148135662079,
+      "learning_rate": 0.00015288529204785363,
+      "loss": 0.4541,
+      "step": 1840
+    },
+    {
+      "epoch": 0.6015523932729625,
+      "grad_norm": 0.32371824979782104,
+      "learning_rate": 0.00015218156228008446,
+      "loss": 0.4287,
+      "step": 1860
+    },
+    {
+      "epoch": 0.6080206985769728,
+      "grad_norm": 0.363788366317749,
+      "learning_rate": 0.00015147783251231528,
+      "loss": 0.428,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6144890038809832,
+      "grad_norm": 0.3932839334011078,
+      "learning_rate": 0.0001507741027445461,
+      "loss": 0.4359,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6209573091849935,
+      "grad_norm": 0.3009307086467743,
+      "learning_rate": 0.00015007037297677694,
+      "loss": 0.3976,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6274256144890039,
+      "grad_norm": 0.3484659194946289,
+      "learning_rate": 0.00014936664320900776,
+      "loss": 0.4584,
+      "step": 1940
+    },
+    {
+      "epoch": 0.6338939197930142,
+      "grad_norm": 0.33728504180908203,
+      "learning_rate": 0.00014866291344123856,
+      "loss": 0.4543,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6403622250970246,
+      "grad_norm": 0.3207024037837982,
+      "learning_rate": 0.0001479591836734694,
+      "loss": 0.4611,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "grad_norm": 0.29134076833724976,
+      "learning_rate": 0.00014725545390570022,
+      "loss": 0.439,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "eval_loss": 0.6003134250640869,
+      "eval_runtime": 968.0302,
+      "eval_samples_per_second": 6.329,
+      "eval_steps_per_second": 0.791,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6532988357050453,
+      "grad_norm": 0.3548874258995056,
+      "learning_rate": 0.00014655172413793104,
+      "loss": 0.4548,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6597671410090556,
+      "grad_norm": 0.3707274794578552,
+      "learning_rate": 0.00014584799437016187,
+      "loss": 0.4271,
+      "step": 2040
+    },
+    {
+      "epoch": 0.666235446313066,
+      "grad_norm": 0.37172695994377136,
+      "learning_rate": 0.0001451442646023927,
+      "loss": 0.4626,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6727037516170763,
+      "grad_norm": 0.3459303081035614,
+      "learning_rate": 0.00014444053483462352,
+      "loss": 0.4317,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6791720569210866,
+      "grad_norm": 0.3477213680744171,
+      "learning_rate": 0.00014373680506685435,
+      "loss": 0.4243,
+      "step": 2100
+    },
+    {
+      "epoch": 0.685640362225097,
+      "grad_norm": 0.31847599148750305,
+      "learning_rate": 0.00014303307529908517,
+      "loss": 0.4234,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6921086675291074,
+      "grad_norm": 0.3231470584869385,
+      "learning_rate": 0.00014232934553131597,
+      "loss": 0.4302,
+      "step": 2140
+    },
+    {
+      "epoch": 0.6985769728331177,
+      "grad_norm": 0.3745759427547455,
+      "learning_rate": 0.0001416256157635468,
+      "loss": 0.4533,
+      "step": 2160
+    },
+    {
+      "epoch": 0.705045278137128,
+      "grad_norm": 0.34834805130958557,
+      "learning_rate": 0.00014092188599577763,
+      "loss": 0.4748,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7115135834411385,
+      "grad_norm": 0.3675827085971832,
+      "learning_rate": 0.00014021815622800845,
+      "loss": 0.4401,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7179818887451488,
+      "grad_norm": 0.2839801013469696,
+      "learning_rate": 0.00013951442646023928,
+      "loss": 0.4332,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7244501940491591,
+      "grad_norm": 0.27037495374679565,
+      "learning_rate": 0.00013881069669247008,
+      "loss": 0.4382,
+      "step": 2240
+    },
+    {
+      "epoch": 0.7309184993531694,
+      "grad_norm": 0.34917208552360535,
+      "learning_rate": 0.00013810696692470093,
+      "loss": 0.4251,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7373868046571799,
+      "grad_norm": 0.39801281690597534,
+      "learning_rate": 0.00013740323715693176,
+      "loss": 0.4432,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7438551099611902,
+      "grad_norm": 0.8932256102561951,
+      "learning_rate": 0.00013669950738916259,
+      "loss": 0.4577,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7503234152652005,
+      "grad_norm": 0.35125109553337097,
+      "learning_rate": 0.00013599577762139339,
+      "loss": 0.4502,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7567917205692108,
+      "grad_norm": 0.3012046813964844,
+      "learning_rate": 0.0001352920478536242,
+      "loss": 0.4513,
+      "step": 2340
+    },
+    {
+      "epoch": 0.7632600258732212,
+      "grad_norm": 0.3242063820362091,
+      "learning_rate": 0.00013458831808585504,
+      "loss": 0.4559,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7697283311772316,
+      "grad_norm": 0.3523860275745392,
+      "learning_rate": 0.00013388458831808587,
+      "loss": 0.4434,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7761966364812419,
+      "grad_norm": 0.29160013794898987,
+      "learning_rate": 0.0001331808585503167,
+      "loss": 0.44,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7826649417852523,
+      "grad_norm": 0.3047971725463867,
+      "learning_rate": 0.0001324771287825475,
+      "loss": 0.4336,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7891332470892626,
+      "grad_norm": 0.3192126750946045,
+      "learning_rate": 0.00013177339901477832,
+      "loss": 0.4495,
+      "step": 2440
+    },
+    {
+      "epoch": 0.795601552393273,
+      "grad_norm": 0.37997299432754517,
+      "learning_rate": 0.00013106966924700917,
+      "loss": 0.4685,
+      "step": 2460
+    },
+    {
+      "epoch": 0.8020698576972833,
+      "grad_norm": 0.2811839282512665,
+      "learning_rate": 0.00013036593947924,
+      "loss": 0.4206,
+      "step": 2480
+    },
+    {
+      "epoch": 0.8085381630012937,
+      "grad_norm": 0.3121369779109955,
+      "learning_rate": 0.0001296622097114708,
+      "loss": 0.425,
+      "step": 2500
+    },
+    {
+      "epoch": 0.815006468305304,
+      "grad_norm": 0.3134746253490448,
+      "learning_rate": 0.00012895847994370162,
+      "loss": 0.4498,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8214747736093143,
+      "grad_norm": 0.36752375960350037,
+      "learning_rate": 0.00012825475017593245,
+      "loss": 0.4288,
+      "step": 2540
+    },
+    {
+      "epoch": 0.8279430789133247,
+      "grad_norm": 0.3915141224861145,
+      "learning_rate": 0.00012755102040816328,
+      "loss": 0.4605,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8344113842173351,
+      "grad_norm": 0.3662431538105011,
+      "learning_rate": 0.00012684729064039408,
+      "loss": 0.4194,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8408796895213454,
+      "grad_norm": 0.3387574851512909,
+      "learning_rate": 0.0001261435608726249,
+      "loss": 0.4258,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8473479948253557,
+      "grad_norm": 0.38025060296058655,
+      "learning_rate": 0.00012543983110485573,
+      "loss": 0.4291,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8538163001293662,
+      "grad_norm": 0.3517441749572754,
+      "learning_rate": 0.00012473610133708658,
+      "loss": 0.4384,
+      "step": 2640
+    },
+    {
+      "epoch": 0.8602846054333765,
+      "grad_norm": 0.33037152886390686,
+      "learning_rate": 0.0001240323715693174,
+      "loss": 0.4452,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8667529107373868,
+      "grad_norm": 0.3645654618740082,
+      "learning_rate": 0.0001233286418015482,
+      "loss": 0.3992,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8732212160413971,
+      "grad_norm": 0.32396814227104187,
+      "learning_rate": 0.00012262491203377904,
+      "loss": 0.4,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8796895213454075,
+      "grad_norm": 0.3377726078033447,
+      "learning_rate": 0.00012192118226600986,
+      "loss": 0.4306,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8861578266494179,
+      "grad_norm": 0.32345983386039734,
+      "learning_rate": 0.00012121745249824067,
+      "loss": 0.4225,
+      "step": 2740
+    },
+    {
+      "epoch": 0.8926261319534282,
+      "grad_norm": 0.34778884053230286,
+      "learning_rate": 0.0001205137227304715,
+      "loss": 0.4256,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8990944372574385,
+      "grad_norm": 0.3861066997051239,
+      "learning_rate": 0.00011980999296270233,
+      "loss": 0.4465,
+      "step": 2780
+    },
+    {
+      "epoch": 0.9055627425614489,
+      "grad_norm": 0.3882784843444824,
+      "learning_rate": 0.00011910626319493314,
+      "loss": 0.4026,
+      "step": 2800
+    },
+    {
+      "epoch": 0.9120310478654593,
+      "grad_norm": 0.4380407929420471,
+      "learning_rate": 0.00011840253342716397,
+      "loss": 0.4375,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9184993531694696,
+      "grad_norm": 0.35943394899368286,
+      "learning_rate": 0.00011769880365939481,
+      "loss": 0.4337,
+      "step": 2840
+    },
+    {
+      "epoch": 0.92496765847348,
+      "grad_norm": 0.3862978219985962,
+      "learning_rate": 0.00011699507389162563,
+      "loss": 0.3949,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9314359637774903,
+      "grad_norm": 0.34592369198799133,
+      "learning_rate": 0.00011629134412385645,
+      "loss": 0.3798,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9379042690815006,
+      "grad_norm": 0.34942877292633057,
+      "learning_rate": 0.00011558761435608727,
+      "loss": 0.4235,
+      "step": 2900
+    },
+    {
+      "epoch": 0.944372574385511,
+      "grad_norm": 0.28018903732299805,
+      "learning_rate": 0.00011488388458831809,
+      "loss": 0.4353,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9508408796895214,
+      "grad_norm": 0.3505672812461853,
+      "learning_rate": 0.00011418015482054891,
+      "loss": 0.3954,
+      "step": 2940
+    },
+    {
+      "epoch": 0.9573091849935317,
+      "grad_norm": 0.3264220654964447,
+      "learning_rate": 0.00011347642505277974,
+      "loss": 0.4422,
+      "step": 2960
+    },
+    {
+      "epoch": 0.963777490297542,
+      "grad_norm": 0.36382535099983215,
+      "learning_rate": 0.00011277269528501055,
+      "loss": 0.4116,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "grad_norm": 0.3051494359970093,
+      "learning_rate": 0.00011206896551724138,
+      "loss": 0.4135,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "eval_loss": 0.5850913524627686,
+      "eval_runtime": 967.6256,
+      "eval_samples_per_second": 6.332,
+      "eval_steps_per_second": 0.792,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9767141009055628,
+      "grad_norm": 0.34533271193504333,
+      "learning_rate": 0.00011136523574947222,
+      "loss": 0.4312,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9831824062095731,
+      "grad_norm": 0.35221609473228455,
+      "learning_rate": 0.00011066150598170305,
+      "loss": 0.4245,
+      "step": 3040
+    },
+    {
+      "epoch": 0.9896507115135834,
+      "grad_norm": 0.3893713057041168,
+      "learning_rate": 0.00010995777621393386,
+      "loss": 0.4048,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9961190168175937,
+      "grad_norm": 0.3586278557777405,
+      "learning_rate": 0.00010925404644616468,
+      "loss": 0.4159,
+      "step": 3080
+    },
+    {
+      "epoch": 1.0025873221216042,
+      "grad_norm": 0.32620444893836975,
+      "learning_rate": 0.0001085503166783955,
+      "loss": 0.3741,
+      "step": 3100
+    },
+    {
+      "epoch": 1.0090556274256144,
+      "grad_norm": 0.33812645077705383,
+      "learning_rate": 0.00010784658691062632,
+      "loss": 0.3694,
+      "step": 3120
+    },
+    {
+      "epoch": 1.0155239327296248,
+      "grad_norm": 0.39038708806037903,
+      "learning_rate": 0.00010714285714285715,
+      "loss": 0.3882,
+      "step": 3140
+    },
+    {
+      "epoch": 1.0219922380336353,
+      "grad_norm": 0.3209969997406006,
+      "learning_rate": 0.00010643912737508796,
+      "loss": 0.3437,
+      "step": 3160
+    },
+    {
+      "epoch": 1.0284605433376455,
+      "grad_norm": 0.3705078661441803,
+      "learning_rate": 0.00010573539760731879,
+      "loss": 0.3568,
+      "step": 3180
+    },
+    {
+      "epoch": 1.034928848641656,
+      "grad_norm": 0.3562203645706177,
+      "learning_rate": 0.0001050316678395496,
+      "loss": 0.3916,
+      "step": 3200
+    },
+    {
+      "epoch": 1.0413971539456663,
+      "grad_norm": 0.3724074363708496,
+      "learning_rate": 0.00010432793807178046,
+      "loss": 0.3866,
+      "step": 3220
+    },
+    {
+      "epoch": 1.0478654592496766,
+      "grad_norm": 0.40453559160232544,
+      "learning_rate": 0.00010362420830401127,
+      "loss": 0.368,
+      "step": 3240
+    },
+    {
+      "epoch": 1.054333764553687,
+      "grad_norm": 0.3316144049167633,
+      "learning_rate": 0.0001029204785362421,
+      "loss": 0.3696,
+      "step": 3260
+    },
+    {
+      "epoch": 1.0608020698576972,
+      "grad_norm": 0.3503650724887848,
+      "learning_rate": 0.00010221674876847291,
+      "loss": 0.3644,
+      "step": 3280
+    },
+    {
+      "epoch": 1.0672703751617076,
+      "grad_norm": 0.3136923909187317,
+      "learning_rate": 0.00010151301900070374,
+      "loss": 0.3414,
+      "step": 3300
+    },
+    {
+      "epoch": 1.073738680465718,
+      "grad_norm": 0.3573864996433258,
+      "learning_rate": 0.00010080928923293456,
+      "loss": 0.3586,
+      "step": 3320
+    },
+    {
+      "epoch": 1.0802069857697283,
+      "grad_norm": 0.26215964555740356,
+      "learning_rate": 0.00010010555946516538,
+      "loss": 0.356,
+      "step": 3340
+    },
+    {
+      "epoch": 1.0866752910737387,
+      "grad_norm": 0.3103466331958771,
+      "learning_rate": 9.940182969739622e-05,
+      "loss": 0.3478,
+      "step": 3360
+    },
+    {
+      "epoch": 1.093143596377749,
+      "grad_norm": 0.3601842224597931,
+      "learning_rate": 9.869809992962703e-05,
+      "loss": 0.3635,
+      "step": 3380
+    },
+    {
+      "epoch": 1.0996119016817594,
+      "grad_norm": 0.3715918958187103,
+      "learning_rate": 9.799437016185785e-05,
+      "loss": 0.3489,
+      "step": 3400
+    },
+    {
+      "epoch": 1.1060802069857698,
+      "grad_norm": 0.4216213524341583,
+      "learning_rate": 9.729064039408867e-05,
+      "loss": 0.3461,
+      "step": 3420
+    },
+    {
+      "epoch": 1.11254851228978,
+      "grad_norm": 0.3878704905509949,
+      "learning_rate": 9.65869106263195e-05,
+      "loss": 0.3548,
+      "step": 3440
+    },
+    {
+      "epoch": 1.1190168175937905,
+      "grad_norm": 0.36658766865730286,
+      "learning_rate": 9.588318085855032e-05,
+      "loss": 0.3395,
+      "step": 3460
+    },
+    {
+      "epoch": 1.1254851228978007,
+      "grad_norm": 0.38134855031967163,
+      "learning_rate": 9.517945109078115e-05,
+      "loss": 0.3829,
+      "step": 3480
+    },
+    {
+      "epoch": 1.131953428201811,
+      "grad_norm": 0.42683905363082886,
+      "learning_rate": 9.447572132301197e-05,
+      "loss": 0.3914,
+      "step": 3500
+    },
+    {
+      "epoch": 1.1384217335058215,
+      "grad_norm": 0.3340086042881012,
+      "learning_rate": 9.377199155524279e-05,
+      "loss": 0.3301,
+      "step": 3520
+    },
+    {
+      "epoch": 1.1448900388098318,
+      "grad_norm": 0.37897104024887085,
+      "learning_rate": 9.306826178747361e-05,
+      "loss": 0.3536,
+      "step": 3540
+    },
+    {
+      "epoch": 1.1513583441138422,
+      "grad_norm": 0.361061692237854,
+      "learning_rate": 9.236453201970444e-05,
+      "loss": 0.338,
+      "step": 3560
+    },
+    {
+      "epoch": 1.1578266494178524,
+      "grad_norm": 0.34730997681617737,
+      "learning_rate": 9.166080225193527e-05,
+      "loss": 0.3679,
+      "step": 3580
+    },
+    {
+      "epoch": 1.1642949547218628,
+      "grad_norm": 0.29455122351646423,
+      "learning_rate": 9.095707248416608e-05,
+      "loss": 0.3602,
+      "step": 3600
+    },
+    {
+      "epoch": 1.1707632600258733,
+      "grad_norm": 0.33159056305885315,
+      "learning_rate": 9.02533427163969e-05,
+      "loss": 0.3359,
+      "step": 3620
+    },
+    {
+      "epoch": 1.1772315653298835,
+      "grad_norm": 0.48726096749305725,
+      "learning_rate": 8.954961294862773e-05,
+      "loss": 0.3879,
+      "step": 3640
+    },
+    {
+      "epoch": 1.183699870633894,
+      "grad_norm": 0.3182481825351715,
+      "learning_rate": 8.884588318085856e-05,
+      "loss": 0.3392,
+      "step": 3660
+    },
+    {
+      "epoch": 1.1901681759379044,
+      "grad_norm": 0.3488766551017761,
+      "learning_rate": 8.814215341308939e-05,
+      "loss": 0.3419,
+      "step": 3680
+    },
+    {
+      "epoch": 1.1966364812419146,
+      "grad_norm": 0.34345540404319763,
+      "learning_rate": 8.74384236453202e-05,
+      "loss": 0.3598,
+      "step": 3700
+    },
+    {
+      "epoch": 1.203104786545925,
+      "grad_norm": 0.37642374634742737,
+      "learning_rate": 8.673469387755102e-05,
+      "loss": 0.3578,
+      "step": 3720
+    },
+    {
+      "epoch": 1.2095730918499352,
+      "grad_norm": 0.3726005554199219,
+      "learning_rate": 8.603096410978185e-05,
+      "loss": 0.3547,
+      "step": 3740
+    },
+    {
+      "epoch": 1.2160413971539457,
+      "grad_norm": 0.36603832244873047,
+      "learning_rate": 8.532723434201268e-05,
+      "loss": 0.3831,
+      "step": 3760
+    },
+    {
+      "epoch": 1.222509702457956,
+      "grad_norm": 0.345103919506073,
+      "learning_rate": 8.462350457424349e-05,
+      "loss": 0.3563,
+      "step": 3780
+    },
+    {
+      "epoch": 1.2289780077619663,
+      "grad_norm": 0.31071504950523376,
+      "learning_rate": 8.391977480647432e-05,
+      "loss": 0.3263,
+      "step": 3800
+    },
+    {
+      "epoch": 1.2354463130659767,
+      "grad_norm": 0.3534263074398041,
+      "learning_rate": 8.321604503870513e-05,
+      "loss": 0.3591,
+      "step": 3820
+    },
+    {
+      "epoch": 1.2419146183699872,
+      "grad_norm": 0.3568720817565918,
+      "learning_rate": 8.251231527093597e-05,
+      "loss": 0.3815,
+      "step": 3840
+    },
+    {
+      "epoch": 1.2483829236739974,
+      "grad_norm": 0.3450963795185089,
+      "learning_rate": 8.18085855031668e-05,
+      "loss": 0.3481,
+      "step": 3860
+    },
+    {
+      "epoch": 1.2548512289780078,
+      "grad_norm": 0.3554648160934448,
+      "learning_rate": 8.110485573539761e-05,
+      "loss": 0.3364,
+      "step": 3880
+    },
+    {
+      "epoch": 1.261319534282018,
+      "grad_norm": 0.4188801348209381,
+      "learning_rate": 8.040112596762844e-05,
+      "loss": 0.3686,
+      "step": 3900
+    },
+    {
+      "epoch": 1.2677878395860285,
+      "grad_norm": 0.3129253089427948,
+      "learning_rate": 7.969739619985925e-05,
+      "loss": 0.3664,
+      "step": 3920
+    },
+    {
+      "epoch": 1.274256144890039,
+      "grad_norm": 0.3885388970375061,
+      "learning_rate": 7.899366643209009e-05,
+      "loss": 0.3415,
+      "step": 3940
+    },
+    {
+      "epoch": 1.2807244501940491,
+      "grad_norm": 0.4131453335285187,
+      "learning_rate": 7.82899366643209e-05,
+      "loss": 0.3552,
+      "step": 3960
+    },
+    {
+      "epoch": 1.2871927554980596,
+      "grad_norm": 0.39486396312713623,
+      "learning_rate": 7.758620689655173e-05,
+      "loss": 0.366,
+      "step": 3980
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "grad_norm": 0.3580794930458069,
+      "learning_rate": 7.688247712878254e-05,
+      "loss": 0.3585,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "eval_loss": 0.5952740907669067,
+      "eval_runtime": 967.8131,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 4000
+    },
+    {
+      "epoch": 1.3001293661060802,
+      "grad_norm": 0.3903474509716034,
+      "learning_rate": 7.617874736101337e-05,
+      "loss": 0.3347,
+      "step": 4020
+    },
+    {
+      "epoch": 1.3065976714100906,
+      "grad_norm": 0.4154898226261139,
+      "learning_rate": 7.54750175932442e-05,
+      "loss": 0.3314,
+      "step": 4040
+    },
+    {
+      "epoch": 1.3130659767141009,
+      "grad_norm": 0.33763521909713745,
+      "learning_rate": 7.477128782547502e-05,
+      "loss": 0.3552,
+      "step": 4060
+    },
+    {
+      "epoch": 1.3195342820181113,
+      "grad_norm": 0.5618635416030884,
+      "learning_rate": 7.406755805770585e-05,
+      "loss": 0.363,
+      "step": 4080
+    },
+    {
+      "epoch": 1.3260025873221215,
+      "grad_norm": 0.37114137411117554,
+      "learning_rate": 7.336382828993666e-05,
+      "loss": 0.3421,
+      "step": 4100
+    },
+    {
+      "epoch": 1.332470892626132,
+      "grad_norm": 0.34323641657829285,
+      "learning_rate": 7.266009852216749e-05,
+      "loss": 0.3659,
+      "step": 4120
+    },
+    {
+      "epoch": 1.3389391979301424,
+      "grad_norm": 0.3527199625968933,
+      "learning_rate": 7.195636875439831e-05,
+      "loss": 0.3412,
+      "step": 4140
+    },
+    {
+      "epoch": 1.3454075032341526,
+      "grad_norm": 0.3413406014442444,
+      "learning_rate": 7.125263898662914e-05,
+      "loss": 0.338,
+      "step": 4160
+    },
+    {
+      "epoch": 1.351875808538163,
+      "grad_norm": 0.34787651896476746,
+      "learning_rate": 7.054890921885995e-05,
+      "loss": 0.3681,
+      "step": 4180
+    },
+    {
+      "epoch": 1.3583441138421732,
+      "grad_norm": 0.2709845006465912,
+      "learning_rate": 6.984517945109078e-05,
+      "loss": 0.325,
+      "step": 4200
+    },
+    {
+      "epoch": 1.3648124191461837,
+      "grad_norm": 0.30966514348983765,
+      "learning_rate": 6.91414496833216e-05,
+      "loss": 0.3677,
+      "step": 4220
+    },
+    {
+      "epoch": 1.371280724450194,
+      "grad_norm": 0.35980409383773804,
+      "learning_rate": 6.843771991555243e-05,
+      "loss": 0.335,
+      "step": 4240
+    },
+    {
+      "epoch": 1.3777490297542043,
+      "grad_norm": 0.5173650979995728,
+      "learning_rate": 6.773399014778326e-05,
+      "loss": 0.348,
+      "step": 4260
+    },
+    {
+      "epoch": 1.3842173350582148,
+      "grad_norm": 0.3048713207244873,
+      "learning_rate": 6.703026038001407e-05,
+      "loss": 0.3323,
+      "step": 4280
+    },
+    {
+      "epoch": 1.390685640362225,
+      "grad_norm": 0.3427985906600952,
+      "learning_rate": 6.63265306122449e-05,
+      "loss": 0.3571,
+      "step": 4300
+    },
+    {
+      "epoch": 1.3971539456662354,
+      "grad_norm": 0.406012624502182,
+      "learning_rate": 6.562280084447573e-05,
+      "loss": 0.3546,
+      "step": 4320
+    },
+    {
+      "epoch": 1.4036222509702458,
+      "grad_norm": 0.31981325149536133,
+      "learning_rate": 6.491907107670655e-05,
+      "loss": 0.3281,
+      "step": 4340
+    },
+    {
+      "epoch": 1.4100905562742563,
+      "grad_norm": 0.3936704099178314,
+      "learning_rate": 6.421534130893737e-05,
+      "loss": 0.3444,
+      "step": 4360
+    },
+    {
+      "epoch": 1.4165588615782665,
+      "grad_norm": 0.3658936023712158,
+      "learning_rate": 6.351161154116819e-05,
+      "loss": 0.339,
+      "step": 4380
+    },
+    {
+      "epoch": 1.4230271668822767,
+      "grad_norm": 0.32523271441459656,
+      "learning_rate": 6.280788177339902e-05,
+      "loss": 0.3311,
+      "step": 4400
+    },
+    {
+      "epoch": 1.4294954721862871,
+      "grad_norm": 0.4120417833328247,
+      "learning_rate": 6.210415200562984e-05,
+      "loss": 0.3342,
+      "step": 4420
+    },
+    {
+      "epoch": 1.4359637774902976,
+      "grad_norm": 0.35293978452682495,
+      "learning_rate": 6.140042223786067e-05,
+      "loss": 0.3561,
+      "step": 4440
+    },
+    {
+      "epoch": 1.442432082794308,
+      "grad_norm": 0.33231040835380554,
+      "learning_rate": 6.0696692470091484e-05,
+      "loss": 0.3573,
+      "step": 4460
+    },
+    {
+      "epoch": 1.4489003880983182,
+      "grad_norm": 0.4345926344394684,
+      "learning_rate": 5.999296270232231e-05,
+      "loss": 0.3364,
+      "step": 4480
+    },
+    {
+      "epoch": 1.4553686934023287,
+      "grad_norm": 0.34958288073539734,
+      "learning_rate": 5.928923293455313e-05,
+      "loss": 0.3162,
+      "step": 4500
+    },
+    {
+      "epoch": 1.4618369987063389,
+      "grad_norm": 0.4669477045536041,
+      "learning_rate": 5.8585503166783964e-05,
+      "loss": 0.3455,
+      "step": 4520
+    },
+    {
+      "epoch": 1.4683053040103493,
+      "grad_norm": 0.32660993933677673,
+      "learning_rate": 5.788177339901478e-05,
+      "loss": 0.3228,
+      "step": 4540
+    },
+    {
+      "epoch": 1.4747736093143597,
+      "grad_norm": 0.3654051423072815,
+      "learning_rate": 5.71780436312456e-05,
+      "loss": 0.3293,
+      "step": 4560
+    },
+    {
+      "epoch": 1.48124191461837,
+      "grad_norm": 0.3496916592121124,
+      "learning_rate": 5.647431386347642e-05,
+      "loss": 0.3382,
+      "step": 4580
+    },
+    {
+      "epoch": 1.4877102199223804,
+      "grad_norm": 0.3890388309955597,
+      "learning_rate": 5.577058409570724e-05,
+      "loss": 0.3468,
+      "step": 4600
+    },
+    {
+      "epoch": 1.4941785252263906,
+      "grad_norm": 0.3924092650413513,
+      "learning_rate": 5.5066854327938076e-05,
+      "loss": 0.3467,
+      "step": 4620
+    },
+    {
+      "epoch": 1.500646830530401,
+      "grad_norm": 0.4015033543109894,
+      "learning_rate": 5.4363124560168896e-05,
+      "loss": 0.3478,
+      "step": 4640
+    },
+    {
+      "epoch": 1.5071151358344115,
+      "grad_norm": 0.34742093086242676,
+      "learning_rate": 5.365939479239972e-05,
+      "loss": 0.3252,
+      "step": 4660
+    },
+    {
+      "epoch": 1.5135834411384217,
+      "grad_norm": 0.3477989435195923,
+      "learning_rate": 5.295566502463054e-05,
+      "loss": 0.3627,
+      "step": 4680
+    },
+    {
+      "epoch": 1.5200517464424321,
+      "grad_norm": 0.4194832146167755,
+      "learning_rate": 5.2251935256861375e-05,
+      "loss": 0.3403,
+      "step": 4700
+    },
+    {
+      "epoch": 1.5265200517464423,
+      "grad_norm": 0.37723594903945923,
+      "learning_rate": 5.1548205489092195e-05,
+      "loss": 0.3424,
+      "step": 4720
+    },
+    {
+      "epoch": 1.5329883570504528,
+      "grad_norm": 0.3695043921470642,
+      "learning_rate": 5.0844475721323015e-05,
+      "loss": 0.3339,
+      "step": 4740
+    },
+    {
+      "epoch": 1.5394566623544632,
+      "grad_norm": 0.35015031695365906,
+      "learning_rate": 5.0140745953553834e-05,
+      "loss": 0.3262,
+      "step": 4760
+    },
+    {
+      "epoch": 1.5459249676584734,
+      "grad_norm": 0.39946597814559937,
+      "learning_rate": 4.943701618578466e-05,
+      "loss": 0.3649,
+      "step": 4780
+    },
+    {
+      "epoch": 1.5523932729624839,
+      "grad_norm": 0.3246670961380005,
+      "learning_rate": 4.873328641801548e-05,
+      "loss": 0.3241,
+      "step": 4800
+    },
+    {
+      "epoch": 1.558861578266494,
+      "grad_norm": 0.41913464665412903,
+      "learning_rate": 4.802955665024631e-05,
+      "loss": 0.3486,
+      "step": 4820
+    },
+    {
+      "epoch": 1.5653298835705045,
+      "grad_norm": 0.38162580132484436,
+      "learning_rate": 4.7325826882477134e-05,
+      "loss": 0.3571,
+      "step": 4840
+    },
+    {
+      "epoch": 1.571798188874515,
+      "grad_norm": 0.34426695108413696,
+      "learning_rate": 4.6622097114707953e-05,
+      "loss": 0.3232,
+      "step": 4860
+    },
+    {
+      "epoch": 1.5782664941785254,
+      "grad_norm": 0.3782813847064972,
+      "learning_rate": 4.591836734693878e-05,
+      "loss": 0.346,
+      "step": 4880
+    },
+    {
+      "epoch": 1.5847347994825356,
+      "grad_norm": 0.42736709117889404,
+      "learning_rate": 4.52146375791696e-05,
+      "loss": 0.3275,
+      "step": 4900
+    },
+    {
+      "epoch": 1.5912031047865458,
+      "grad_norm": 0.37077978253364563,
+      "learning_rate": 4.4510907811400426e-05,
+      "loss": 0.3312,
+      "step": 4920
+    },
+    {
+      "epoch": 1.5976714100905562,
+      "grad_norm": 0.3196958303451538,
+      "learning_rate": 4.3807178043631246e-05,
+      "loss": 0.3156,
+      "step": 4940
+    },
+    {
+      "epoch": 1.6041397153945667,
+      "grad_norm": 0.36680254340171814,
+      "learning_rate": 4.3103448275862066e-05,
+      "loss": 0.3293,
+      "step": 4960
+    },
+    {
+      "epoch": 1.6106080206985771,
+      "grad_norm": 0.44455257058143616,
+      "learning_rate": 4.239971850809289e-05,
+      "loss": 0.3512,
+      "step": 4980
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "grad_norm": 0.37418290972709656,
+      "learning_rate": 4.169598874032372e-05,
+      "loss": 0.3422,
+      "step": 5000
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "eval_loss": 0.5881822109222412,
+      "eval_runtime": 967.8912,
+      "eval_samples_per_second": 6.33,
+      "eval_steps_per_second": 0.791,
+      "step": 5000
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.669845911961469e+18,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

ultival_mistral/checkpoint-5000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb8e5db938f70ff2ddfe787ade649d14bb55cc2be7a65bd63a20c3303bf68492
+size 5368

ultival_mistral/checkpoint-6000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Ministral-8B-Instruct-2410
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

ultival_mistral/checkpoint-6000/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Ministral-8B-Instruct-2410",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "gate_proj",
+    "up_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

ultival_mistral/checkpoint-6000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f082f6d0a3a97b05923cb2b2b2fbe02ce9f81b4b6db500a43193fc1828f6cdff
+size 349243752

ultival_mistral/checkpoint-6000/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% if messages[1]['role'] == 'user' %}{{ '[INST] ' + messages[0]['content'] + ' ' + messages[1]['content'] + ' [/INST]' }}{% set loop_messages = messages[2:] %}{% else %}{{ '[INST] ' + messages[0]['content'] + ' [/INST]' }}{% set loop_messages = messages[1:] %}{% endif %}{% else %}{% set loop_messages = messages %}{% endif %}{% for message in loop_messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}

ultival_mistral/checkpoint-6000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f55137f8a9b9997a59aaeb82cd818e104ce757097a31612eb87abed422dce57b
+size 177919028

ultival_mistral/checkpoint-6000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cf932362a907148c2c99f7826e21fa7280b5016d990a6358e60ec3cec98b016
+size 14244

ultival_mistral/checkpoint-6000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ed7383692d087a8f053c6ea044fdaeeeda6d990035145da8af603021621b27
+size 1064

ultival_mistral/checkpoint-6000/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

ultival_mistral/checkpoint-6000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be2965d3e435dbdef9376e5bdb3dcc2bfc7e271aa5152ae896cf53c7e920adf8
+size 17078235

ultival_mistral/checkpoint-6000/trainer_state.json

@@ -0,0 +1,2182 @@
+{
+  "best_global_step": 6000,
+  "best_metric": 0.5839738845825195,
+  "best_model_checkpoint": "./ultival_run2/checkpoint-6000",
+  "epoch": 1.9404915912031049,
+  "eval_steps": 1000,
+  "global_step": 6000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00646830530401035,
+      "grad_norm": 0.8976479172706604,
+      "learning_rate": 7.6e-06,
+      "loss": 1.5544,
+      "step": 20
+    },
+    {
+      "epoch": 0.0129366106080207,
+      "grad_norm": 0.726784348487854,
+      "learning_rate": 1.56e-05,
+      "loss": 1.242,
+      "step": 40
+    },
+    {
+      "epoch": 0.019404915912031046,
+      "grad_norm": 0.4917074143886566,
+      "learning_rate": 2.36e-05,
+      "loss": 0.6694,
+      "step": 60
+    },
+    {
+      "epoch": 0.0258732212160414,
+      "grad_norm": 0.3471176326274872,
+      "learning_rate": 3.16e-05,
+      "loss": 0.5334,
+      "step": 80
+    },
+    {
+      "epoch": 0.03234152652005175,
+      "grad_norm": 0.41759616136550903,
+      "learning_rate": 3.960000000000001e-05,
+      "loss": 0.5435,
+      "step": 100
+    },
+    {
+      "epoch": 0.03880983182406209,
+      "grad_norm": 0.3755040466785431,
+      "learning_rate": 4.76e-05,
+      "loss": 0.5684,
+      "step": 120
+    },
+    {
+      "epoch": 0.045278137128072445,
+      "grad_norm": 0.37058988213539124,
+      "learning_rate": 5.560000000000001e-05,
+      "loss": 0.5329,
+      "step": 140
+    },
+    {
+      "epoch": 0.0517464424320828,
+      "grad_norm": 0.42603132128715515,
+      "learning_rate": 6.36e-05,
+      "loss": 0.5457,
+      "step": 160
+    },
+    {
+      "epoch": 0.05821474773609314,
+      "grad_norm": 0.3154866099357605,
+      "learning_rate": 7.16e-05,
+      "loss": 0.5472,
+      "step": 180
+    },
+    {
+      "epoch": 0.0646830530401035,
+      "grad_norm": 0.3919835388660431,
+      "learning_rate": 7.960000000000001e-05,
+      "loss": 0.4675,
+      "step": 200
+    },
+    {
+      "epoch": 0.07115135834411385,
+      "grad_norm": 0.32180657982826233,
+      "learning_rate": 8.76e-05,
+      "loss": 0.5078,
+      "step": 220
+    },
+    {
+      "epoch": 0.07761966364812418,
+      "grad_norm": 0.3246982991695404,
+      "learning_rate": 9.56e-05,
+      "loss": 0.488,
+      "step": 240
+    },
+    {
+      "epoch": 0.08408796895213454,
+      "grad_norm": 0.341487318277359,
+      "learning_rate": 0.00010360000000000001,
+      "loss": 0.5058,
+      "step": 260
+    },
+    {
+      "epoch": 0.09055627425614489,
+      "grad_norm": 0.3597952127456665,
+      "learning_rate": 0.00011160000000000002,
+      "loss": 0.4574,
+      "step": 280
+    },
+    {
+      "epoch": 0.09702457956015524,
+      "grad_norm": 0.40912091732025146,
+      "learning_rate": 0.00011960000000000001,
+      "loss": 0.4931,
+      "step": 300
+    },
+    {
+      "epoch": 0.1034928848641656,
+      "grad_norm": 0.37430697679519653,
+      "learning_rate": 0.0001276,
+      "loss": 0.4724,
+      "step": 320
+    },
+    {
+      "epoch": 0.10996119016817593,
+      "grad_norm": 0.3920972943305969,
+      "learning_rate": 0.00013560000000000002,
+      "loss": 0.4997,
+      "step": 340
+    },
+    {
+      "epoch": 0.11642949547218628,
+      "grad_norm": 0.32511982321739197,
+      "learning_rate": 0.0001436,
+      "loss": 0.4889,
+      "step": 360
+    },
+    {
+      "epoch": 0.12289780077619664,
+      "grad_norm": 0.3926583528518677,
+      "learning_rate": 0.0001516,
+      "loss": 0.4772,
+      "step": 380
+    },
+    {
+      "epoch": 0.129366106080207,
+      "grad_norm": 0.3535909950733185,
+      "learning_rate": 0.0001596,
+      "loss": 0.465,
+      "step": 400
+    },
+    {
+      "epoch": 0.13583441138421734,
+      "grad_norm": 0.34381625056266785,
+      "learning_rate": 0.0001676,
+      "loss": 0.4934,
+      "step": 420
+    },
+    {
+      "epoch": 0.1423027166882277,
+      "grad_norm": 0.3753703236579895,
+      "learning_rate": 0.0001756,
+      "loss": 0.5068,
+      "step": 440
+    },
+    {
+      "epoch": 0.14877102199223805,
+      "grad_norm": 0.34895479679107666,
+      "learning_rate": 0.00018360000000000002,
+      "loss": 0.5186,
+      "step": 460
+    },
+    {
+      "epoch": 0.15523932729624837,
+      "grad_norm": 0.24720332026481628,
+      "learning_rate": 0.0001916,
+      "loss": 0.461,
+      "step": 480
+    },
+    {
+      "epoch": 0.16170763260025872,
+      "grad_norm": 0.3713696002960205,
+      "learning_rate": 0.0001996,
+      "loss": 0.4932,
+      "step": 500
+    },
+    {
+      "epoch": 0.16817593790426907,
+      "grad_norm": 0.3724994659423828,
+      "learning_rate": 0.00019933145672061927,
+      "loss": 0.5036,
+      "step": 520
+    },
+    {
+      "epoch": 0.17464424320827943,
+      "grad_norm": 0.3562956750392914,
+      "learning_rate": 0.00019862772695285012,
+      "loss": 0.4871,
+      "step": 540
+    },
+    {
+      "epoch": 0.18111254851228978,
+      "grad_norm": 0.331737756729126,
+      "learning_rate": 0.00019792399718508095,
+      "loss": 0.5178,
+      "step": 560
+    },
+    {
+      "epoch": 0.18758085381630013,
+      "grad_norm": 0.34049859642982483,
+      "learning_rate": 0.00019722026741731178,
+      "loss": 0.488,
+      "step": 580
+    },
+    {
+      "epoch": 0.19404915912031048,
+      "grad_norm": 0.36076661944389343,
+      "learning_rate": 0.00019651653764954258,
+      "loss": 0.4821,
+      "step": 600
+    },
+    {
+      "epoch": 0.20051746442432083,
+      "grad_norm": 0.3059399127960205,
+      "learning_rate": 0.0001958128078817734,
+      "loss": 0.4924,
+      "step": 620
+    },
+    {
+      "epoch": 0.2069857697283312,
+      "grad_norm": 0.30692920088768005,
+      "learning_rate": 0.00019510907811400423,
+      "loss": 0.4822,
+      "step": 640
+    },
+    {
+      "epoch": 0.21345407503234154,
+      "grad_norm": 0.3319448232650757,
+      "learning_rate": 0.00019440534834623506,
+      "loss": 0.4607,
+      "step": 660
+    },
+    {
+      "epoch": 0.21992238033635186,
+      "grad_norm": 0.3047769069671631,
+      "learning_rate": 0.00019370161857846588,
+      "loss": 0.5357,
+      "step": 680
+    },
+    {
+      "epoch": 0.22639068564036222,
+      "grad_norm": 0.4353258013725281,
+      "learning_rate": 0.00019299788881069668,
+      "loss": 0.4738,
+      "step": 700
+    },
+    {
+      "epoch": 0.23285899094437257,
+      "grad_norm": 0.30877336859703064,
+      "learning_rate": 0.00019229415904292754,
+      "loss": 0.5128,
+      "step": 720
+    },
+    {
+      "epoch": 0.23932729624838292,
+      "grad_norm": 0.3188530206680298,
+      "learning_rate": 0.00019159042927515836,
+      "loss": 0.4651,
+      "step": 740
+    },
+    {
+      "epoch": 0.24579560155239327,
+      "grad_norm": 0.40678778290748596,
+      "learning_rate": 0.0001908866995073892,
+      "loss": 0.5112,
+      "step": 760
+    },
+    {
+      "epoch": 0.2522639068564036,
+      "grad_norm": 0.34188327193260193,
+      "learning_rate": 0.00019018296973962,
+      "loss": 0.487,
+      "step": 780
+    },
+    {
+      "epoch": 0.258732212160414,
+      "grad_norm": 0.3002854585647583,
+      "learning_rate": 0.00018947923997185081,
+      "loss": 0.512,
+      "step": 800
+    },
+    {
+      "epoch": 0.2652005174644243,
+      "grad_norm": 0.35756272077560425,
+      "learning_rate": 0.00018877551020408164,
+      "loss": 0.4812,
+      "step": 820
+    },
+    {
+      "epoch": 0.2716688227684347,
+      "grad_norm": 0.33527258038520813,
+      "learning_rate": 0.00018807178043631247,
+      "loss": 0.4895,
+      "step": 840
+    },
+    {
+      "epoch": 0.278137128072445,
+      "grad_norm": 0.3472883105278015,
+      "learning_rate": 0.0001873680506685433,
+      "loss": 0.4671,
+      "step": 860
+    },
+    {
+      "epoch": 0.2846054333764554,
+      "grad_norm": 0.3604671061038971,
+      "learning_rate": 0.0001866643209007741,
+      "loss": 0.5233,
+      "step": 880
+    },
+    {
+      "epoch": 0.2910737386804657,
+      "grad_norm": 0.31021153926849365,
+      "learning_rate": 0.00018596059113300492,
+      "loss": 0.491,
+      "step": 900
+    },
+    {
+      "epoch": 0.2975420439844761,
+      "grad_norm": 0.6417067050933838,
+      "learning_rate": 0.00018525686136523577,
+      "loss": 0.4858,
+      "step": 920
+    },
+    {
+      "epoch": 0.3040103492884864,
+      "grad_norm": 0.33719781041145325,
+      "learning_rate": 0.0001845531315974666,
+      "loss": 0.4722,
+      "step": 940
+    },
+    {
+      "epoch": 0.31047865459249674,
+      "grad_norm": 0.3568169176578522,
+      "learning_rate": 0.0001838494018296974,
+      "loss": 0.5088,
+      "step": 960
+    },
+    {
+      "epoch": 0.3169469598965071,
+      "grad_norm": 0.42162615060806274,
+      "learning_rate": 0.00018314567206192823,
+      "loss": 0.5091,
+      "step": 980
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "grad_norm": 0.3212260603904724,
+      "learning_rate": 0.00018244194229415905,
+      "loss": 0.4654,
+      "step": 1000
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "eval_loss": 0.6167479753494263,
+      "eval_runtime": 1013.3659,
+      "eval_samples_per_second": 6.046,
+      "eval_steps_per_second": 0.756,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3298835705045278,
+      "grad_norm": 0.3184376060962677,
+      "learning_rate": 0.00018173821252638988,
+      "loss": 0.4712,
+      "step": 1020
+    },
+    {
+      "epoch": 0.33635187580853815,
+      "grad_norm": 0.3114645183086395,
+      "learning_rate": 0.0001810344827586207,
+      "loss": 0.4393,
+      "step": 1040
+    },
+    {
+      "epoch": 0.3428201811125485,
+      "grad_norm": 0.30122488737106323,
+      "learning_rate": 0.0001803307529908515,
+      "loss": 0.4882,
+      "step": 1060
+    },
+    {
+      "epoch": 0.34928848641655885,
+      "grad_norm": 0.2877393066883087,
+      "learning_rate": 0.00017962702322308233,
+      "loss": 0.413,
+      "step": 1080
+    },
+    {
+      "epoch": 0.35575679172056923,
+      "grad_norm": 0.33227887749671936,
+      "learning_rate": 0.00017892329345531319,
+      "loss": 0.4527,
+      "step": 1100
+    },
+    {
+      "epoch": 0.36222509702457956,
+      "grad_norm": 0.33423128724098206,
+      "learning_rate": 0.000178219563687544,
+      "loss": 0.4604,
+      "step": 1120
+    },
+    {
+      "epoch": 0.36869340232858994,
+      "grad_norm": 0.36922487616539,
+      "learning_rate": 0.0001775158339197748,
+      "loss": 0.5044,
+      "step": 1140
+    },
+    {
+      "epoch": 0.37516170763260026,
+      "grad_norm": 0.34816262125968933,
+      "learning_rate": 0.00017681210415200564,
+      "loss": 0.4898,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3816300129366106,
+      "grad_norm": 0.32086044549942017,
+      "learning_rate": 0.00017610837438423646,
+      "loss": 0.4647,
+      "step": 1180
+    },
+    {
+      "epoch": 0.38809831824062097,
+      "grad_norm": 0.3879236876964569,
+      "learning_rate": 0.0001754046446164673,
+      "loss": 0.5179,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3945666235446313,
+      "grad_norm": 0.2899722158908844,
+      "learning_rate": 0.00017470091484869812,
+      "loss": 0.4786,
+      "step": 1220
+    },
+    {
+      "epoch": 0.40103492884864167,
+      "grad_norm": 0.3844458758831024,
+      "learning_rate": 0.00017399718508092892,
+      "loss": 0.4513,
+      "step": 1240
+    },
+    {
+      "epoch": 0.407503234152652,
+      "grad_norm": 0.3678169548511505,
+      "learning_rate": 0.00017329345531315974,
+      "loss": 0.4789,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4139715394566624,
+      "grad_norm": 0.37138116359710693,
+      "learning_rate": 0.00017258972554539057,
+      "loss": 0.4694,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4204398447606727,
+      "grad_norm": 0.3604221045970917,
+      "learning_rate": 0.00017188599577762142,
+      "loss": 0.4496,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4269081500646831,
+      "grad_norm": 0.33961811661720276,
+      "learning_rate": 0.00017118226600985222,
+      "loss": 0.4696,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4333764553686934,
+      "grad_norm": 0.3244430124759674,
+      "learning_rate": 0.00017047853624208305,
+      "loss": 0.4322,
+      "step": 1340
+    },
+    {
+      "epoch": 0.4398447606727037,
+      "grad_norm": 0.2879267930984497,
+      "learning_rate": 0.00016977480647431388,
+      "loss": 0.4646,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4463130659767141,
+      "grad_norm": 0.3437345027923584,
+      "learning_rate": 0.0001690710767065447,
+      "loss": 0.4711,
+      "step": 1380
+    },
+    {
+      "epoch": 0.45278137128072443,
+      "grad_norm": 0.3166882395744324,
+      "learning_rate": 0.00016836734693877553,
+      "loss": 0.486,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4592496765847348,
+      "grad_norm": 0.3677555322647095,
+      "learning_rate": 0.00016766361717100633,
+      "loss": 0.4677,
+      "step": 1420
+    },
+    {
+      "epoch": 0.46571798188874514,
+      "grad_norm": 0.32657769322395325,
+      "learning_rate": 0.00016695988740323715,
+      "loss": 0.4649,
+      "step": 1440
+    },
+    {
+      "epoch": 0.4721862871927555,
+      "grad_norm": 0.2980731427669525,
+      "learning_rate": 0.00016625615763546798,
+      "loss": 0.4547,
+      "step": 1460
+    },
+    {
+      "epoch": 0.47865459249676584,
+      "grad_norm": 0.3690549433231354,
+      "learning_rate": 0.0001655524278676988,
+      "loss": 0.4512,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4851228978007762,
+      "grad_norm": 0.31456947326660156,
+      "learning_rate": 0.00016484869809992963,
+      "loss": 0.4675,
+      "step": 1500
+    },
+    {
+      "epoch": 0.49159120310478654,
+      "grad_norm": 0.3823802173137665,
+      "learning_rate": 0.00016414496833216046,
+      "loss": 0.4849,
+      "step": 1520
+    },
+    {
+      "epoch": 0.49805950840879687,
+      "grad_norm": 0.33280324935913086,
+      "learning_rate": 0.0001634412385643913,
+      "loss": 0.4876,
+      "step": 1540
+    },
+    {
+      "epoch": 0.5045278137128072,
+      "grad_norm": 0.28788062930107117,
+      "learning_rate": 0.00016273750879662211,
+      "loss": 0.426,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5109961190168176,
+      "grad_norm": 0.3809375762939453,
+      "learning_rate": 0.00016203377902885294,
+      "loss": 0.4676,
+      "step": 1580
+    },
+    {
+      "epoch": 0.517464424320828,
+      "grad_norm": 0.27940312027931213,
+      "learning_rate": 0.00016133004926108374,
+      "loss": 0.4413,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5239327296248383,
+      "grad_norm": 0.30946919322013855,
+      "learning_rate": 0.00016062631949331457,
+      "loss": 0.4395,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5304010349288486,
+      "grad_norm": 0.36931583285331726,
+      "learning_rate": 0.0001599225897255454,
+      "loss": 0.4565,
+      "step": 1640
+    },
+    {
+      "epoch": 0.536869340232859,
+      "grad_norm": 0.3652470111846924,
+      "learning_rate": 0.00015921885995777622,
+      "loss": 0.4522,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5433376455368694,
+      "grad_norm": 0.3060782551765442,
+      "learning_rate": 0.00015851513019000705,
+      "loss": 0.45,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5498059508408797,
+      "grad_norm": 0.3275803029537201,
+      "learning_rate": 0.00015781140042223787,
+      "loss": 0.445,
+      "step": 1700
+    },
+    {
+      "epoch": 0.55627425614489,
+      "grad_norm": 0.34865352511405945,
+      "learning_rate": 0.0001571076706544687,
+      "loss": 0.4306,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5627425614489003,
+      "grad_norm": 0.42525383830070496,
+      "learning_rate": 0.00015640394088669953,
+      "loss": 0.4544,
+      "step": 1740
+    },
+    {
+      "epoch": 0.5692108667529108,
+      "grad_norm": 0.32165050506591797,
+      "learning_rate": 0.00015570021111893035,
+      "loss": 0.484,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5756791720569211,
+      "grad_norm": 0.3213440477848053,
+      "learning_rate": 0.00015499648135116115,
+      "loss": 0.4469,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5821474773609314,
+      "grad_norm": 0.42835432291030884,
+      "learning_rate": 0.00015429275158339198,
+      "loss": 0.4592,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5886157826649417,
+      "grad_norm": 0.3861497938632965,
+      "learning_rate": 0.0001535890218156228,
+      "loss": 0.4607,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5950840879689522,
+      "grad_norm": 0.4156148135662079,
+      "learning_rate": 0.00015288529204785363,
+      "loss": 0.4541,
+      "step": 1840
+    },
+    {
+      "epoch": 0.6015523932729625,
+      "grad_norm": 0.32371824979782104,
+      "learning_rate": 0.00015218156228008446,
+      "loss": 0.4287,
+      "step": 1860
+    },
+    {
+      "epoch": 0.6080206985769728,
+      "grad_norm": 0.363788366317749,
+      "learning_rate": 0.00015147783251231528,
+      "loss": 0.428,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6144890038809832,
+      "grad_norm": 0.3932839334011078,
+      "learning_rate": 0.0001507741027445461,
+      "loss": 0.4359,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6209573091849935,
+      "grad_norm": 0.3009307086467743,
+      "learning_rate": 0.00015007037297677694,
+      "loss": 0.3976,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6274256144890039,
+      "grad_norm": 0.3484659194946289,
+      "learning_rate": 0.00014936664320900776,
+      "loss": 0.4584,
+      "step": 1940
+    },
+    {
+      "epoch": 0.6338939197930142,
+      "grad_norm": 0.33728504180908203,
+      "learning_rate": 0.00014866291344123856,
+      "loss": 0.4543,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6403622250970246,
+      "grad_norm": 0.3207024037837982,
+      "learning_rate": 0.0001479591836734694,
+      "loss": 0.4611,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "grad_norm": 0.29134076833724976,
+      "learning_rate": 0.00014725545390570022,
+      "loss": 0.439,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "eval_loss": 0.6003134250640869,
+      "eval_runtime": 968.0302,
+      "eval_samples_per_second": 6.329,
+      "eval_steps_per_second": 0.791,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6532988357050453,
+      "grad_norm": 0.3548874258995056,
+      "learning_rate": 0.00014655172413793104,
+      "loss": 0.4548,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6597671410090556,
+      "grad_norm": 0.3707274794578552,
+      "learning_rate": 0.00014584799437016187,
+      "loss": 0.4271,
+      "step": 2040
+    },
+    {
+      "epoch": 0.666235446313066,
+      "grad_norm": 0.37172695994377136,
+      "learning_rate": 0.0001451442646023927,
+      "loss": 0.4626,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6727037516170763,
+      "grad_norm": 0.3459303081035614,
+      "learning_rate": 0.00014444053483462352,
+      "loss": 0.4317,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6791720569210866,
+      "grad_norm": 0.3477213680744171,
+      "learning_rate": 0.00014373680506685435,
+      "loss": 0.4243,
+      "step": 2100
+    },
+    {
+      "epoch": 0.685640362225097,
+      "grad_norm": 0.31847599148750305,
+      "learning_rate": 0.00014303307529908517,
+      "loss": 0.4234,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6921086675291074,
+      "grad_norm": 0.3231470584869385,
+      "learning_rate": 0.00014232934553131597,
+      "loss": 0.4302,
+      "step": 2140
+    },
+    {
+      "epoch": 0.6985769728331177,
+      "grad_norm": 0.3745759427547455,
+      "learning_rate": 0.0001416256157635468,
+      "loss": 0.4533,
+      "step": 2160
+    },
+    {
+      "epoch": 0.705045278137128,
+      "grad_norm": 0.34834805130958557,
+      "learning_rate": 0.00014092188599577763,
+      "loss": 0.4748,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7115135834411385,
+      "grad_norm": 0.3675827085971832,
+      "learning_rate": 0.00014021815622800845,
+      "loss": 0.4401,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7179818887451488,
+      "grad_norm": 0.2839801013469696,
+      "learning_rate": 0.00013951442646023928,
+      "loss": 0.4332,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7244501940491591,
+      "grad_norm": 0.27037495374679565,
+      "learning_rate": 0.00013881069669247008,
+      "loss": 0.4382,
+      "step": 2240
+    },
+    {
+      "epoch": 0.7309184993531694,
+      "grad_norm": 0.34917208552360535,
+      "learning_rate": 0.00013810696692470093,
+      "loss": 0.4251,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7373868046571799,
+      "grad_norm": 0.39801281690597534,
+      "learning_rate": 0.00013740323715693176,
+      "loss": 0.4432,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7438551099611902,
+      "grad_norm": 0.8932256102561951,
+      "learning_rate": 0.00013669950738916259,
+      "loss": 0.4577,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7503234152652005,
+      "grad_norm": 0.35125109553337097,
+      "learning_rate": 0.00013599577762139339,
+      "loss": 0.4502,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7567917205692108,
+      "grad_norm": 0.3012046813964844,
+      "learning_rate": 0.0001352920478536242,
+      "loss": 0.4513,
+      "step": 2340
+    },
+    {
+      "epoch": 0.7632600258732212,
+      "grad_norm": 0.3242063820362091,
+      "learning_rate": 0.00013458831808585504,
+      "loss": 0.4559,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7697283311772316,
+      "grad_norm": 0.3523860275745392,
+      "learning_rate": 0.00013388458831808587,
+      "loss": 0.4434,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7761966364812419,
+      "grad_norm": 0.29160013794898987,
+      "learning_rate": 0.0001331808585503167,
+      "loss": 0.44,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7826649417852523,
+      "grad_norm": 0.3047971725463867,
+      "learning_rate": 0.0001324771287825475,
+      "loss": 0.4336,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7891332470892626,
+      "grad_norm": 0.3192126750946045,
+      "learning_rate": 0.00013177339901477832,
+      "loss": 0.4495,
+      "step": 2440
+    },
+    {
+      "epoch": 0.795601552393273,
+      "grad_norm": 0.37997299432754517,
+      "learning_rate": 0.00013106966924700917,
+      "loss": 0.4685,
+      "step": 2460
+    },
+    {
+      "epoch": 0.8020698576972833,
+      "grad_norm": 0.2811839282512665,
+      "learning_rate": 0.00013036593947924,
+      "loss": 0.4206,
+      "step": 2480
+    },
+    {
+      "epoch": 0.8085381630012937,
+      "grad_norm": 0.3121369779109955,
+      "learning_rate": 0.0001296622097114708,
+      "loss": 0.425,
+      "step": 2500
+    },
+    {
+      "epoch": 0.815006468305304,
+      "grad_norm": 0.3134746253490448,
+      "learning_rate": 0.00012895847994370162,
+      "loss": 0.4498,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8214747736093143,
+      "grad_norm": 0.36752375960350037,
+      "learning_rate": 0.00012825475017593245,
+      "loss": 0.4288,
+      "step": 2540
+    },
+    {
+      "epoch": 0.8279430789133247,
+      "grad_norm": 0.3915141224861145,
+      "learning_rate": 0.00012755102040816328,
+      "loss": 0.4605,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8344113842173351,
+      "grad_norm": 0.3662431538105011,
+      "learning_rate": 0.00012684729064039408,
+      "loss": 0.4194,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8408796895213454,
+      "grad_norm": 0.3387574851512909,
+      "learning_rate": 0.0001261435608726249,
+      "loss": 0.4258,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8473479948253557,
+      "grad_norm": 0.38025060296058655,
+      "learning_rate": 0.00012543983110485573,
+      "loss": 0.4291,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8538163001293662,
+      "grad_norm": 0.3517441749572754,
+      "learning_rate": 0.00012473610133708658,
+      "loss": 0.4384,
+      "step": 2640
+    },
+    {
+      "epoch": 0.8602846054333765,
+      "grad_norm": 0.33037152886390686,
+      "learning_rate": 0.0001240323715693174,
+      "loss": 0.4452,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8667529107373868,
+      "grad_norm": 0.3645654618740082,
+      "learning_rate": 0.0001233286418015482,
+      "loss": 0.3992,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8732212160413971,
+      "grad_norm": 0.32396814227104187,
+      "learning_rate": 0.00012262491203377904,
+      "loss": 0.4,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8796895213454075,
+      "grad_norm": 0.3377726078033447,
+      "learning_rate": 0.00012192118226600986,
+      "loss": 0.4306,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8861578266494179,
+      "grad_norm": 0.32345983386039734,
+      "learning_rate": 0.00012121745249824067,
+      "loss": 0.4225,
+      "step": 2740
+    },
+    {
+      "epoch": 0.8926261319534282,
+      "grad_norm": 0.34778884053230286,
+      "learning_rate": 0.0001205137227304715,
+      "loss": 0.4256,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8990944372574385,
+      "grad_norm": 0.3861066997051239,
+      "learning_rate": 0.00011980999296270233,
+      "loss": 0.4465,
+      "step": 2780
+    },
+    {
+      "epoch": 0.9055627425614489,
+      "grad_norm": 0.3882784843444824,
+      "learning_rate": 0.00011910626319493314,
+      "loss": 0.4026,
+      "step": 2800
+    },
+    {
+      "epoch": 0.9120310478654593,
+      "grad_norm": 0.4380407929420471,
+      "learning_rate": 0.00011840253342716397,
+      "loss": 0.4375,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9184993531694696,
+      "grad_norm": 0.35943394899368286,
+      "learning_rate": 0.00011769880365939481,
+      "loss": 0.4337,
+      "step": 2840
+    },
+    {
+      "epoch": 0.92496765847348,
+      "grad_norm": 0.3862978219985962,
+      "learning_rate": 0.00011699507389162563,
+      "loss": 0.3949,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9314359637774903,
+      "grad_norm": 0.34592369198799133,
+      "learning_rate": 0.00011629134412385645,
+      "loss": 0.3798,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9379042690815006,
+      "grad_norm": 0.34942877292633057,
+      "learning_rate": 0.00011558761435608727,
+      "loss": 0.4235,
+      "step": 2900
+    },
+    {
+      "epoch": 0.944372574385511,
+      "grad_norm": 0.28018903732299805,
+      "learning_rate": 0.00011488388458831809,
+      "loss": 0.4353,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9508408796895214,
+      "grad_norm": 0.3505672812461853,
+      "learning_rate": 0.00011418015482054891,
+      "loss": 0.3954,
+      "step": 2940
+    },
+    {
+      "epoch": 0.9573091849935317,
+      "grad_norm": 0.3264220654964447,
+      "learning_rate": 0.00011347642505277974,
+      "loss": 0.4422,
+      "step": 2960
+    },
+    {
+      "epoch": 0.963777490297542,
+      "grad_norm": 0.36382535099983215,
+      "learning_rate": 0.00011277269528501055,
+      "loss": 0.4116,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "grad_norm": 0.3051494359970093,
+      "learning_rate": 0.00011206896551724138,
+      "loss": 0.4135,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "eval_loss": 0.5850913524627686,
+      "eval_runtime": 967.6256,
+      "eval_samples_per_second": 6.332,
+      "eval_steps_per_second": 0.792,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9767141009055628,
+      "grad_norm": 0.34533271193504333,
+      "learning_rate": 0.00011136523574947222,
+      "loss": 0.4312,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9831824062095731,
+      "grad_norm": 0.35221609473228455,
+      "learning_rate": 0.00011066150598170305,
+      "loss": 0.4245,
+      "step": 3040
+    },
+    {
+      "epoch": 0.9896507115135834,
+      "grad_norm": 0.3893713057041168,
+      "learning_rate": 0.00010995777621393386,
+      "loss": 0.4048,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9961190168175937,
+      "grad_norm": 0.3586278557777405,
+      "learning_rate": 0.00010925404644616468,
+      "loss": 0.4159,
+      "step": 3080
+    },
+    {
+      "epoch": 1.0025873221216042,
+      "grad_norm": 0.32620444893836975,
+      "learning_rate": 0.0001085503166783955,
+      "loss": 0.3741,
+      "step": 3100
+    },
+    {
+      "epoch": 1.0090556274256144,
+      "grad_norm": 0.33812645077705383,
+      "learning_rate": 0.00010784658691062632,
+      "loss": 0.3694,
+      "step": 3120
+    },
+    {
+      "epoch": 1.0155239327296248,
+      "grad_norm": 0.39038708806037903,
+      "learning_rate": 0.00010714285714285715,
+      "loss": 0.3882,
+      "step": 3140
+    },
+    {
+      "epoch": 1.0219922380336353,
+      "grad_norm": 0.3209969997406006,
+      "learning_rate": 0.00010643912737508796,
+      "loss": 0.3437,
+      "step": 3160
+    },
+    {
+      "epoch": 1.0284605433376455,
+      "grad_norm": 0.3705078661441803,
+      "learning_rate": 0.00010573539760731879,
+      "loss": 0.3568,
+      "step": 3180
+    },
+    {
+      "epoch": 1.034928848641656,
+      "grad_norm": 0.3562203645706177,
+      "learning_rate": 0.0001050316678395496,
+      "loss": 0.3916,
+      "step": 3200
+    },
+    {
+      "epoch": 1.0413971539456663,
+      "grad_norm": 0.3724074363708496,
+      "learning_rate": 0.00010432793807178046,
+      "loss": 0.3866,
+      "step": 3220
+    },
+    {
+      "epoch": 1.0478654592496766,
+      "grad_norm": 0.40453559160232544,
+      "learning_rate": 0.00010362420830401127,
+      "loss": 0.368,
+      "step": 3240
+    },
+    {
+      "epoch": 1.054333764553687,
+      "grad_norm": 0.3316144049167633,
+      "learning_rate": 0.0001029204785362421,
+      "loss": 0.3696,
+      "step": 3260
+    },
+    {
+      "epoch": 1.0608020698576972,
+      "grad_norm": 0.3503650724887848,
+      "learning_rate": 0.00010221674876847291,
+      "loss": 0.3644,
+      "step": 3280
+    },
+    {
+      "epoch": 1.0672703751617076,
+      "grad_norm": 0.3136923909187317,
+      "learning_rate": 0.00010151301900070374,
+      "loss": 0.3414,
+      "step": 3300
+    },
+    {
+      "epoch": 1.073738680465718,
+      "grad_norm": 0.3573864996433258,
+      "learning_rate": 0.00010080928923293456,
+      "loss": 0.3586,
+      "step": 3320
+    },
+    {
+      "epoch": 1.0802069857697283,
+      "grad_norm": 0.26215964555740356,
+      "learning_rate": 0.00010010555946516538,
+      "loss": 0.356,
+      "step": 3340
+    },
+    {
+      "epoch": 1.0866752910737387,
+      "grad_norm": 0.3103466331958771,
+      "learning_rate": 9.940182969739622e-05,
+      "loss": 0.3478,
+      "step": 3360
+    },
+    {
+      "epoch": 1.093143596377749,
+      "grad_norm": 0.3601842224597931,
+      "learning_rate": 9.869809992962703e-05,
+      "loss": 0.3635,
+      "step": 3380
+    },
+    {
+      "epoch": 1.0996119016817594,
+      "grad_norm": 0.3715918958187103,
+      "learning_rate": 9.799437016185785e-05,
+      "loss": 0.3489,
+      "step": 3400
+    },
+    {
+      "epoch": 1.1060802069857698,
+      "grad_norm": 0.4216213524341583,
+      "learning_rate": 9.729064039408867e-05,
+      "loss": 0.3461,
+      "step": 3420
+    },
+    {
+      "epoch": 1.11254851228978,
+      "grad_norm": 0.3878704905509949,
+      "learning_rate": 9.65869106263195e-05,
+      "loss": 0.3548,
+      "step": 3440
+    },
+    {
+      "epoch": 1.1190168175937905,
+      "grad_norm": 0.36658766865730286,
+      "learning_rate": 9.588318085855032e-05,
+      "loss": 0.3395,
+      "step": 3460
+    },
+    {
+      "epoch": 1.1254851228978007,
+      "grad_norm": 0.38134855031967163,
+      "learning_rate": 9.517945109078115e-05,
+      "loss": 0.3829,
+      "step": 3480
+    },
+    {
+      "epoch": 1.131953428201811,
+      "grad_norm": 0.42683905363082886,
+      "learning_rate": 9.447572132301197e-05,
+      "loss": 0.3914,
+      "step": 3500
+    },
+    {
+      "epoch": 1.1384217335058215,
+      "grad_norm": 0.3340086042881012,
+      "learning_rate": 9.377199155524279e-05,
+      "loss": 0.3301,
+      "step": 3520
+    },
+    {
+      "epoch": 1.1448900388098318,
+      "grad_norm": 0.37897104024887085,
+      "learning_rate": 9.306826178747361e-05,
+      "loss": 0.3536,
+      "step": 3540
+    },
+    {
+      "epoch": 1.1513583441138422,
+      "grad_norm": 0.361061692237854,
+      "learning_rate": 9.236453201970444e-05,
+      "loss": 0.338,
+      "step": 3560
+    },
+    {
+      "epoch": 1.1578266494178524,
+      "grad_norm": 0.34730997681617737,
+      "learning_rate": 9.166080225193527e-05,
+      "loss": 0.3679,
+      "step": 3580
+    },
+    {
+      "epoch": 1.1642949547218628,
+      "grad_norm": 0.29455122351646423,
+      "learning_rate": 9.095707248416608e-05,
+      "loss": 0.3602,
+      "step": 3600
+    },
+    {
+      "epoch": 1.1707632600258733,
+      "grad_norm": 0.33159056305885315,
+      "learning_rate": 9.02533427163969e-05,
+      "loss": 0.3359,
+      "step": 3620
+    },
+    {
+      "epoch": 1.1772315653298835,
+      "grad_norm": 0.48726096749305725,
+      "learning_rate": 8.954961294862773e-05,
+      "loss": 0.3879,
+      "step": 3640
+    },
+    {
+      "epoch": 1.183699870633894,
+      "grad_norm": 0.3182481825351715,
+      "learning_rate": 8.884588318085856e-05,
+      "loss": 0.3392,
+      "step": 3660
+    },
+    {
+      "epoch": 1.1901681759379044,
+      "grad_norm": 0.3488766551017761,
+      "learning_rate": 8.814215341308939e-05,
+      "loss": 0.3419,
+      "step": 3680
+    },
+    {
+      "epoch": 1.1966364812419146,
+      "grad_norm": 0.34345540404319763,
+      "learning_rate": 8.74384236453202e-05,
+      "loss": 0.3598,
+      "step": 3700
+    },
+    {
+      "epoch": 1.203104786545925,
+      "grad_norm": 0.37642374634742737,
+      "learning_rate": 8.673469387755102e-05,
+      "loss": 0.3578,
+      "step": 3720
+    },
+    {
+      "epoch": 1.2095730918499352,
+      "grad_norm": 0.3726005554199219,
+      "learning_rate": 8.603096410978185e-05,
+      "loss": 0.3547,
+      "step": 3740
+    },
+    {
+      "epoch": 1.2160413971539457,
+      "grad_norm": 0.36603832244873047,
+      "learning_rate": 8.532723434201268e-05,
+      "loss": 0.3831,
+      "step": 3760
+    },
+    {
+      "epoch": 1.222509702457956,
+      "grad_norm": 0.345103919506073,
+      "learning_rate": 8.462350457424349e-05,
+      "loss": 0.3563,
+      "step": 3780
+    },
+    {
+      "epoch": 1.2289780077619663,
+      "grad_norm": 0.31071504950523376,
+      "learning_rate": 8.391977480647432e-05,
+      "loss": 0.3263,
+      "step": 3800
+    },
+    {
+      "epoch": 1.2354463130659767,
+      "grad_norm": 0.3534263074398041,
+      "learning_rate": 8.321604503870513e-05,
+      "loss": 0.3591,
+      "step": 3820
+    },
+    {
+      "epoch": 1.2419146183699872,
+      "grad_norm": 0.3568720817565918,
+      "learning_rate": 8.251231527093597e-05,
+      "loss": 0.3815,
+      "step": 3840
+    },
+    {
+      "epoch": 1.2483829236739974,
+      "grad_norm": 0.3450963795185089,
+      "learning_rate": 8.18085855031668e-05,
+      "loss": 0.3481,
+      "step": 3860
+    },
+    {
+      "epoch": 1.2548512289780078,
+      "grad_norm": 0.3554648160934448,
+      "learning_rate": 8.110485573539761e-05,
+      "loss": 0.3364,
+      "step": 3880
+    },
+    {
+      "epoch": 1.261319534282018,
+      "grad_norm": 0.4188801348209381,
+      "learning_rate": 8.040112596762844e-05,
+      "loss": 0.3686,
+      "step": 3900
+    },
+    {
+      "epoch": 1.2677878395860285,
+      "grad_norm": 0.3129253089427948,
+      "learning_rate": 7.969739619985925e-05,
+      "loss": 0.3664,
+      "step": 3920
+    },
+    {
+      "epoch": 1.274256144890039,
+      "grad_norm": 0.3885388970375061,
+      "learning_rate": 7.899366643209009e-05,
+      "loss": 0.3415,
+      "step": 3940
+    },
+    {
+      "epoch": 1.2807244501940491,
+      "grad_norm": 0.4131453335285187,
+      "learning_rate": 7.82899366643209e-05,
+      "loss": 0.3552,
+      "step": 3960
+    },
+    {
+      "epoch": 1.2871927554980596,
+      "grad_norm": 0.39486396312713623,
+      "learning_rate": 7.758620689655173e-05,
+      "loss": 0.366,
+      "step": 3980
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "grad_norm": 0.3580794930458069,
+      "learning_rate": 7.688247712878254e-05,
+      "loss": 0.3585,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "eval_loss": 0.5952740907669067,
+      "eval_runtime": 967.8131,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 4000
+    },
+    {
+      "epoch": 1.3001293661060802,
+      "grad_norm": 0.3903474509716034,
+      "learning_rate": 7.617874736101337e-05,
+      "loss": 0.3347,
+      "step": 4020
+    },
+    {
+      "epoch": 1.3065976714100906,
+      "grad_norm": 0.4154898226261139,
+      "learning_rate": 7.54750175932442e-05,
+      "loss": 0.3314,
+      "step": 4040
+    },
+    {
+      "epoch": 1.3130659767141009,
+      "grad_norm": 0.33763521909713745,
+      "learning_rate": 7.477128782547502e-05,
+      "loss": 0.3552,
+      "step": 4060
+    },
+    {
+      "epoch": 1.3195342820181113,
+      "grad_norm": 0.5618635416030884,
+      "learning_rate": 7.406755805770585e-05,
+      "loss": 0.363,
+      "step": 4080
+    },
+    {
+      "epoch": 1.3260025873221215,
+      "grad_norm": 0.37114137411117554,
+      "learning_rate": 7.336382828993666e-05,
+      "loss": 0.3421,
+      "step": 4100
+    },
+    {
+      "epoch": 1.332470892626132,
+      "grad_norm": 0.34323641657829285,
+      "learning_rate": 7.266009852216749e-05,
+      "loss": 0.3659,
+      "step": 4120
+    },
+    {
+      "epoch": 1.3389391979301424,
+      "grad_norm": 0.3527199625968933,
+      "learning_rate": 7.195636875439831e-05,
+      "loss": 0.3412,
+      "step": 4140
+    },
+    {
+      "epoch": 1.3454075032341526,
+      "grad_norm": 0.3413406014442444,
+      "learning_rate": 7.125263898662914e-05,
+      "loss": 0.338,
+      "step": 4160
+    },
+    {
+      "epoch": 1.351875808538163,
+      "grad_norm": 0.34787651896476746,
+      "learning_rate": 7.054890921885995e-05,
+      "loss": 0.3681,
+      "step": 4180
+    },
+    {
+      "epoch": 1.3583441138421732,
+      "grad_norm": 0.2709845006465912,
+      "learning_rate": 6.984517945109078e-05,
+      "loss": 0.325,
+      "step": 4200
+    },
+    {
+      "epoch": 1.3648124191461837,
+      "grad_norm": 0.30966514348983765,
+      "learning_rate": 6.91414496833216e-05,
+      "loss": 0.3677,
+      "step": 4220
+    },
+    {
+      "epoch": 1.371280724450194,
+      "grad_norm": 0.35980409383773804,
+      "learning_rate": 6.843771991555243e-05,
+      "loss": 0.335,
+      "step": 4240
+    },
+    {
+      "epoch": 1.3777490297542043,
+      "grad_norm": 0.5173650979995728,
+      "learning_rate": 6.773399014778326e-05,
+      "loss": 0.348,
+      "step": 4260
+    },
+    {
+      "epoch": 1.3842173350582148,
+      "grad_norm": 0.3048713207244873,
+      "learning_rate": 6.703026038001407e-05,
+      "loss": 0.3323,
+      "step": 4280
+    },
+    {
+      "epoch": 1.390685640362225,
+      "grad_norm": 0.3427985906600952,
+      "learning_rate": 6.63265306122449e-05,
+      "loss": 0.3571,
+      "step": 4300
+    },
+    {
+      "epoch": 1.3971539456662354,
+      "grad_norm": 0.406012624502182,
+      "learning_rate": 6.562280084447573e-05,
+      "loss": 0.3546,
+      "step": 4320
+    },
+    {
+      "epoch": 1.4036222509702458,
+      "grad_norm": 0.31981325149536133,
+      "learning_rate": 6.491907107670655e-05,
+      "loss": 0.3281,
+      "step": 4340
+    },
+    {
+      "epoch": 1.4100905562742563,
+      "grad_norm": 0.3936704099178314,
+      "learning_rate": 6.421534130893737e-05,
+      "loss": 0.3444,
+      "step": 4360
+    },
+    {
+      "epoch": 1.4165588615782665,
+      "grad_norm": 0.3658936023712158,
+      "learning_rate": 6.351161154116819e-05,
+      "loss": 0.339,
+      "step": 4380
+    },
+    {
+      "epoch": 1.4230271668822767,
+      "grad_norm": 0.32523271441459656,
+      "learning_rate": 6.280788177339902e-05,
+      "loss": 0.3311,
+      "step": 4400
+    },
+    {
+      "epoch": 1.4294954721862871,
+      "grad_norm": 0.4120417833328247,
+      "learning_rate": 6.210415200562984e-05,
+      "loss": 0.3342,
+      "step": 4420
+    },
+    {
+      "epoch": 1.4359637774902976,
+      "grad_norm": 0.35293978452682495,
+      "learning_rate": 6.140042223786067e-05,
+      "loss": 0.3561,
+      "step": 4440
+    },
+    {
+      "epoch": 1.442432082794308,
+      "grad_norm": 0.33231040835380554,
+      "learning_rate": 6.0696692470091484e-05,
+      "loss": 0.3573,
+      "step": 4460
+    },
+    {
+      "epoch": 1.4489003880983182,
+      "grad_norm": 0.4345926344394684,
+      "learning_rate": 5.999296270232231e-05,
+      "loss": 0.3364,
+      "step": 4480
+    },
+    {
+      "epoch": 1.4553686934023287,
+      "grad_norm": 0.34958288073539734,
+      "learning_rate": 5.928923293455313e-05,
+      "loss": 0.3162,
+      "step": 4500
+    },
+    {
+      "epoch": 1.4618369987063389,
+      "grad_norm": 0.4669477045536041,
+      "learning_rate": 5.8585503166783964e-05,
+      "loss": 0.3455,
+      "step": 4520
+    },
+    {
+      "epoch": 1.4683053040103493,
+      "grad_norm": 0.32660993933677673,
+      "learning_rate": 5.788177339901478e-05,
+      "loss": 0.3228,
+      "step": 4540
+    },
+    {
+      "epoch": 1.4747736093143597,
+      "grad_norm": 0.3654051423072815,
+      "learning_rate": 5.71780436312456e-05,
+      "loss": 0.3293,
+      "step": 4560
+    },
+    {
+      "epoch": 1.48124191461837,
+      "grad_norm": 0.3496916592121124,
+      "learning_rate": 5.647431386347642e-05,
+      "loss": 0.3382,
+      "step": 4580
+    },
+    {
+      "epoch": 1.4877102199223804,
+      "grad_norm": 0.3890388309955597,
+      "learning_rate": 5.577058409570724e-05,
+      "loss": 0.3468,
+      "step": 4600
+    },
+    {
+      "epoch": 1.4941785252263906,
+      "grad_norm": 0.3924092650413513,
+      "learning_rate": 5.5066854327938076e-05,
+      "loss": 0.3467,
+      "step": 4620
+    },
+    {
+      "epoch": 1.500646830530401,
+      "grad_norm": 0.4015033543109894,
+      "learning_rate": 5.4363124560168896e-05,
+      "loss": 0.3478,
+      "step": 4640
+    },
+    {
+      "epoch": 1.5071151358344115,
+      "grad_norm": 0.34742093086242676,
+      "learning_rate": 5.365939479239972e-05,
+      "loss": 0.3252,
+      "step": 4660
+    },
+    {
+      "epoch": 1.5135834411384217,
+      "grad_norm": 0.3477989435195923,
+      "learning_rate": 5.295566502463054e-05,
+      "loss": 0.3627,
+      "step": 4680
+    },
+    {
+      "epoch": 1.5200517464424321,
+      "grad_norm": 0.4194832146167755,
+      "learning_rate": 5.2251935256861375e-05,
+      "loss": 0.3403,
+      "step": 4700
+    },
+    {
+      "epoch": 1.5265200517464423,
+      "grad_norm": 0.37723594903945923,
+      "learning_rate": 5.1548205489092195e-05,
+      "loss": 0.3424,
+      "step": 4720
+    },
+    {
+      "epoch": 1.5329883570504528,
+      "grad_norm": 0.3695043921470642,
+      "learning_rate": 5.0844475721323015e-05,
+      "loss": 0.3339,
+      "step": 4740
+    },
+    {
+      "epoch": 1.5394566623544632,
+      "grad_norm": 0.35015031695365906,
+      "learning_rate": 5.0140745953553834e-05,
+      "loss": 0.3262,
+      "step": 4760
+    },
+    {
+      "epoch": 1.5459249676584734,
+      "grad_norm": 0.39946597814559937,
+      "learning_rate": 4.943701618578466e-05,
+      "loss": 0.3649,
+      "step": 4780
+    },
+    {
+      "epoch": 1.5523932729624839,
+      "grad_norm": 0.3246670961380005,
+      "learning_rate": 4.873328641801548e-05,
+      "loss": 0.3241,
+      "step": 4800
+    },
+    {
+      "epoch": 1.558861578266494,
+      "grad_norm": 0.41913464665412903,
+      "learning_rate": 4.802955665024631e-05,
+      "loss": 0.3486,
+      "step": 4820
+    },
+    {
+      "epoch": 1.5653298835705045,
+      "grad_norm": 0.38162580132484436,
+      "learning_rate": 4.7325826882477134e-05,
+      "loss": 0.3571,
+      "step": 4840
+    },
+    {
+      "epoch": 1.571798188874515,
+      "grad_norm": 0.34426695108413696,
+      "learning_rate": 4.6622097114707953e-05,
+      "loss": 0.3232,
+      "step": 4860
+    },
+    {
+      "epoch": 1.5782664941785254,
+      "grad_norm": 0.3782813847064972,
+      "learning_rate": 4.591836734693878e-05,
+      "loss": 0.346,
+      "step": 4880
+    },
+    {
+      "epoch": 1.5847347994825356,
+      "grad_norm": 0.42736709117889404,
+      "learning_rate": 4.52146375791696e-05,
+      "loss": 0.3275,
+      "step": 4900
+    },
+    {
+      "epoch": 1.5912031047865458,
+      "grad_norm": 0.37077978253364563,
+      "learning_rate": 4.4510907811400426e-05,
+      "loss": 0.3312,
+      "step": 4920
+    },
+    {
+      "epoch": 1.5976714100905562,
+      "grad_norm": 0.3196958303451538,
+      "learning_rate": 4.3807178043631246e-05,
+      "loss": 0.3156,
+      "step": 4940
+    },
+    {
+      "epoch": 1.6041397153945667,
+      "grad_norm": 0.36680254340171814,
+      "learning_rate": 4.3103448275862066e-05,
+      "loss": 0.3293,
+      "step": 4960
+    },
+    {
+      "epoch": 1.6106080206985771,
+      "grad_norm": 0.44455257058143616,
+      "learning_rate": 4.239971850809289e-05,
+      "loss": 0.3512,
+      "step": 4980
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "grad_norm": 0.37418290972709656,
+      "learning_rate": 4.169598874032372e-05,
+      "loss": 0.3422,
+      "step": 5000
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "eval_loss": 0.5881822109222412,
+      "eval_runtime": 967.8912,
+      "eval_samples_per_second": 6.33,
+      "eval_steps_per_second": 0.791,
+      "step": 5000
+    },
+    {
+      "epoch": 1.6235446313065975,
+      "grad_norm": 0.3209955096244812,
+      "learning_rate": 4.0992258972554545e-05,
+      "loss": 0.3158,
+      "step": 5020
+    },
+    {
+      "epoch": 1.630012936610608,
+      "grad_norm": 0.31715771555900574,
+      "learning_rate": 4.0288529204785365e-05,
+      "loss": 0.3167,
+      "step": 5040
+    },
+    {
+      "epoch": 1.6364812419146184,
+      "grad_norm": 0.3968450725078583,
+      "learning_rate": 3.958479943701619e-05,
+      "loss": 0.3219,
+      "step": 5060
+    },
+    {
+      "epoch": 1.6429495472186288,
+      "grad_norm": 0.3187231123447418,
+      "learning_rate": 3.888106966924701e-05,
+      "loss": 0.3288,
+      "step": 5080
+    },
+    {
+      "epoch": 1.649417852522639,
+      "grad_norm": 0.35491153597831726,
+      "learning_rate": 3.817733990147783e-05,
+      "loss": 0.3298,
+      "step": 5100
+    },
+    {
+      "epoch": 1.6558861578266493,
+      "grad_norm": 0.4044147729873657,
+      "learning_rate": 3.747361013370866e-05,
+      "loss": 0.3505,
+      "step": 5120
+    },
+    {
+      "epoch": 1.6623544631306597,
+      "grad_norm": 0.33881333470344543,
+      "learning_rate": 3.676988036593948e-05,
+      "loss": 0.3269,
+      "step": 5140
+    },
+    {
+      "epoch": 1.6688227684346701,
+      "grad_norm": 0.3626604378223419,
+      "learning_rate": 3.6066150598170304e-05,
+      "loss": 0.3319,
+      "step": 5160
+    },
+    {
+      "epoch": 1.6752910737386806,
+      "grad_norm": 0.39335256814956665,
+      "learning_rate": 3.536242083040113e-05,
+      "loss": 0.3234,
+      "step": 5180
+    },
+    {
+      "epoch": 1.6817593790426908,
+      "grad_norm": 0.35490670800209045,
+      "learning_rate": 3.465869106263195e-05,
+      "loss": 0.3357,
+      "step": 5200
+    },
+    {
+      "epoch": 1.688227684346701,
+      "grad_norm": 0.303774356842041,
+      "learning_rate": 3.3954961294862776e-05,
+      "loss": 0.3609,
+      "step": 5220
+    },
+    {
+      "epoch": 1.6946959896507114,
+      "grad_norm": 0.3438192307949066,
+      "learning_rate": 3.3251231527093596e-05,
+      "loss": 0.3351,
+      "step": 5240
+    },
+    {
+      "epoch": 1.7011642949547219,
+      "grad_norm": 0.3261934518814087,
+      "learning_rate": 3.254750175932442e-05,
+      "loss": 0.3192,
+      "step": 5260
+    },
+    {
+      "epoch": 1.7076326002587323,
+      "grad_norm": 0.39429745078086853,
+      "learning_rate": 3.184377199155524e-05,
+      "loss": 0.3479,
+      "step": 5280
+    },
+    {
+      "epoch": 1.7141009055627425,
+      "grad_norm": 0.4267764985561371,
+      "learning_rate": 3.114004222378607e-05,
+      "loss": 0.3306,
+      "step": 5300
+    },
+    {
+      "epoch": 1.720569210866753,
+      "grad_norm": 0.3425467908382416,
+      "learning_rate": 3.0436312456016892e-05,
+      "loss": 0.3356,
+      "step": 5320
+    },
+    {
+      "epoch": 1.7270375161707632,
+      "grad_norm": 0.3064535856246948,
+      "learning_rate": 2.9732582688247712e-05,
+      "loss": 0.3341,
+      "step": 5340
+    },
+    {
+      "epoch": 1.7335058214747736,
+      "grad_norm": 0.3956790566444397,
+      "learning_rate": 2.902885292047854e-05,
+      "loss": 0.3375,
+      "step": 5360
+    },
+    {
+      "epoch": 1.739974126778784,
+      "grad_norm": 0.36128687858581543,
+      "learning_rate": 2.8325123152709358e-05,
+      "loss": 0.3395,
+      "step": 5380
+    },
+    {
+      "epoch": 1.7464424320827943,
+      "grad_norm": 0.33011215925216675,
+      "learning_rate": 2.7621393384940185e-05,
+      "loss": 0.3078,
+      "step": 5400
+    },
+    {
+      "epoch": 1.7529107373868047,
+      "grad_norm": 0.3741130530834198,
+      "learning_rate": 2.6917663617171008e-05,
+      "loss": 0.362,
+      "step": 5420
+    },
+    {
+      "epoch": 1.759379042690815,
+      "grad_norm": 0.319671094417572,
+      "learning_rate": 2.6213933849401827e-05,
+      "loss": 0.3105,
+      "step": 5440
+    },
+    {
+      "epoch": 1.7658473479948253,
+      "grad_norm": 0.4425293505191803,
+      "learning_rate": 2.5510204081632654e-05,
+      "loss": 0.3327,
+      "step": 5460
+    },
+    {
+      "epoch": 1.7723156532988358,
+      "grad_norm": 0.43660980463027954,
+      "learning_rate": 2.4806474313863477e-05,
+      "loss": 0.3057,
+      "step": 5480
+    },
+    {
+      "epoch": 1.7787839586028462,
+      "grad_norm": 0.3476182222366333,
+      "learning_rate": 2.41027445460943e-05,
+      "loss": 0.3362,
+      "step": 5500
+    },
+    {
+      "epoch": 1.7852522639068564,
+      "grad_norm": 0.37895113229751587,
+      "learning_rate": 2.3399014778325123e-05,
+      "loss": 0.337,
+      "step": 5520
+    },
+    {
+      "epoch": 1.7917205692108666,
+      "grad_norm": 0.38524091243743896,
+      "learning_rate": 2.2695285010555947e-05,
+      "loss": 0.362,
+      "step": 5540
+    },
+    {
+      "epoch": 1.798188874514877,
+      "grad_norm": 0.38090500235557556,
+      "learning_rate": 2.199155524278677e-05,
+      "loss": 0.3268,
+      "step": 5560
+    },
+    {
+      "epoch": 1.8046571798188875,
+      "grad_norm": 0.4094742238521576,
+      "learning_rate": 2.1287825475017596e-05,
+      "loss": 0.3353,
+      "step": 5580
+    },
+    {
+      "epoch": 1.811125485122898,
+      "grad_norm": 0.37963709235191345,
+      "learning_rate": 2.058409570724842e-05,
+      "loss": 0.3197,
+      "step": 5600
+    },
+    {
+      "epoch": 1.8175937904269082,
+      "grad_norm": 0.4362301230430603,
+      "learning_rate": 1.988036593947924e-05,
+      "loss": 0.3405,
+      "step": 5620
+    },
+    {
+      "epoch": 1.8240620957309184,
+      "grad_norm": 0.3662102520465851,
+      "learning_rate": 1.9176636171710062e-05,
+      "loss": 0.3262,
+      "step": 5640
+    },
+    {
+      "epoch": 1.8305304010349288,
+      "grad_norm": 0.3674371838569641,
+      "learning_rate": 1.847290640394089e-05,
+      "loss": 0.3394,
+      "step": 5660
+    },
+    {
+      "epoch": 1.8369987063389392,
+      "grad_norm": 0.31638914346694946,
+      "learning_rate": 1.7769176636171712e-05,
+      "loss": 0.3072,
+      "step": 5680
+    },
+    {
+      "epoch": 1.8434670116429497,
+      "grad_norm": 0.39859539270401,
+      "learning_rate": 1.7065446868402535e-05,
+      "loss": 0.3398,
+      "step": 5700
+    },
+    {
+      "epoch": 1.84993531694696,
+      "grad_norm": 0.37528204917907715,
+      "learning_rate": 1.6361717100633358e-05,
+      "loss": 0.3362,
+      "step": 5720
+    },
+    {
+      "epoch": 1.85640362225097,
+      "grad_norm": 0.40589115023612976,
+      "learning_rate": 1.565798733286418e-05,
+      "loss": 0.3369,
+      "step": 5740
+    },
+    {
+      "epoch": 1.8628719275549805,
+      "grad_norm": 0.3010273277759552,
+      "learning_rate": 1.4954257565095003e-05,
+      "loss": 0.3293,
+      "step": 5760
+    },
+    {
+      "epoch": 1.869340232858991,
+      "grad_norm": 0.47433990240097046,
+      "learning_rate": 1.4250527797325827e-05,
+      "loss": 0.3137,
+      "step": 5780
+    },
+    {
+      "epoch": 1.8758085381630014,
+      "grad_norm": 0.3495091199874878,
+      "learning_rate": 1.354679802955665e-05,
+      "loss": 0.3077,
+      "step": 5800
+    },
+    {
+      "epoch": 1.8822768434670116,
+      "grad_norm": 0.33797210454940796,
+      "learning_rate": 1.2843068261787475e-05,
+      "loss": 0.3454,
+      "step": 5820
+    },
+    {
+      "epoch": 1.8887451487710218,
+      "grad_norm": 0.34069564938545227,
+      "learning_rate": 1.2139338494018297e-05,
+      "loss": 0.3087,
+      "step": 5840
+    },
+    {
+      "epoch": 1.8952134540750323,
+      "grad_norm": 0.4089032709598541,
+      "learning_rate": 1.1435608726249122e-05,
+      "loss": 0.3487,
+      "step": 5860
+    },
+    {
+      "epoch": 1.9016817593790427,
+      "grad_norm": 0.3203008472919464,
+      "learning_rate": 1.0731878958479945e-05,
+      "loss": 0.3301,
+      "step": 5880
+    },
+    {
+      "epoch": 1.9081500646830531,
+      "grad_norm": 0.3505829870700836,
+      "learning_rate": 1.0028149190710768e-05,
+      "loss": 0.3319,
+      "step": 5900
+    },
+    {
+      "epoch": 1.9146183699870634,
+      "grad_norm": 0.29925537109375,
+      "learning_rate": 9.324419422941591e-06,
+      "loss": 0.3317,
+      "step": 5920
+    },
+    {
+      "epoch": 1.9210866752910736,
+      "grad_norm": 0.4060503840446472,
+      "learning_rate": 8.620689655172414e-06,
+      "loss": 0.3393,
+      "step": 5940
+    },
+    {
+      "epoch": 1.927554980595084,
+      "grad_norm": 0.4205261170864105,
+      "learning_rate": 7.916959887403237e-06,
+      "loss": 0.3375,
+      "step": 5960
+    },
+    {
+      "epoch": 1.9340232858990944,
+      "grad_norm": 0.44757845997810364,
+      "learning_rate": 7.213230119634061e-06,
+      "loss": 0.3329,
+      "step": 5980
+    },
+    {
+      "epoch": 1.9404915912031049,
+      "grad_norm": 0.5326984524726868,
+      "learning_rate": 6.509500351864884e-06,
+      "loss": 0.342,
+      "step": 6000
+    },
+    {
+      "epoch": 1.9404915912031049,
+      "eval_loss": 0.5839738845825195,
+      "eval_runtime": 967.7859,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 6000
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 7.991147541685469e+18,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

ultival_mistral/checkpoint-6000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb8e5db938f70ff2ddfe787ade649d14bb55cc2be7a65bd63a20c3303bf68492
+size 5368

ultival_mistral/checkpoint-6184/README.md

@@ -0,0 +1,202 @@
+---
+base_model: mistralai/Ministral-8B-Instruct-2410
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

ultival_mistral/checkpoint-6184/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Ministral-8B-Instruct-2410",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj",
+    "gate_proj",
+    "up_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

ultival_mistral/checkpoint-6184/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cdcd01ee05b36166eb8d36352e3dd9de01a098cc56f647bf981c0fc5ec37393
+size 349243752

ultival_mistral/checkpoint-6184/chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% if messages[1]['role'] == 'user' %}{{ '[INST] ' + messages[0]['content'] + ' ' + messages[1]['content'] + ' [/INST]' }}{% set loop_messages = messages[2:] %}{% else %}{{ '[INST] ' + messages[0]['content'] + ' [/INST]' }}{% set loop_messages = messages[1:] %}{% endif %}{% else %}{% set loop_messages = messages %}{% endif %}{% for message in loop_messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}

ultival_mistral/checkpoint-6184/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52eac4bb89fa48810acf94bc308f7d720f59643d37ef36715604087b81444897
+size 177919028

ultival_mistral/checkpoint-6184/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cf932362a907148c2c99f7826e21fa7280b5016d990a6358e60ec3cec98b016
+size 14244

ultival_mistral/checkpoint-6184/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd069de76fd3803039b7454910298a611b2a1a5486b5af24595ccb064fdf7323
+size 1064

ultival_mistral/checkpoint-6184/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

ultival_mistral/checkpoint-6184/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be2965d3e435dbdef9376e5bdb3dcc2bfc7e271aa5152ae896cf53c7e920adf8
+size 17078235

ultival_mistral/checkpoint-6184/trainer_state.json

@@ -0,0 +1,2245 @@
+{
+  "best_global_step": 6000,
+  "best_metric": 0.5839738845825195,
+  "best_model_checkpoint": "./ultival_run2/checkpoint-6000",
+  "epoch": 2.0,
+  "eval_steps": 1000,
+  "global_step": 6184,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00646830530401035,
+      "grad_norm": 0.8976479172706604,
+      "learning_rate": 7.6e-06,
+      "loss": 1.5544,
+      "step": 20
+    },
+    {
+      "epoch": 0.0129366106080207,
+      "grad_norm": 0.726784348487854,
+      "learning_rate": 1.56e-05,
+      "loss": 1.242,
+      "step": 40
+    },
+    {
+      "epoch": 0.019404915912031046,
+      "grad_norm": 0.4917074143886566,
+      "learning_rate": 2.36e-05,
+      "loss": 0.6694,
+      "step": 60
+    },
+    {
+      "epoch": 0.0258732212160414,
+      "grad_norm": 0.3471176326274872,
+      "learning_rate": 3.16e-05,
+      "loss": 0.5334,
+      "step": 80
+    },
+    {
+      "epoch": 0.03234152652005175,
+      "grad_norm": 0.41759616136550903,
+      "learning_rate": 3.960000000000001e-05,
+      "loss": 0.5435,
+      "step": 100
+    },
+    {
+      "epoch": 0.03880983182406209,
+      "grad_norm": 0.3755040466785431,
+      "learning_rate": 4.76e-05,
+      "loss": 0.5684,
+      "step": 120
+    },
+    {
+      "epoch": 0.045278137128072445,
+      "grad_norm": 0.37058988213539124,
+      "learning_rate": 5.560000000000001e-05,
+      "loss": 0.5329,
+      "step": 140
+    },
+    {
+      "epoch": 0.0517464424320828,
+      "grad_norm": 0.42603132128715515,
+      "learning_rate": 6.36e-05,
+      "loss": 0.5457,
+      "step": 160
+    },
+    {
+      "epoch": 0.05821474773609314,
+      "grad_norm": 0.3154866099357605,
+      "learning_rate": 7.16e-05,
+      "loss": 0.5472,
+      "step": 180
+    },
+    {
+      "epoch": 0.0646830530401035,
+      "grad_norm": 0.3919835388660431,
+      "learning_rate": 7.960000000000001e-05,
+      "loss": 0.4675,
+      "step": 200
+    },
+    {
+      "epoch": 0.07115135834411385,
+      "grad_norm": 0.32180657982826233,
+      "learning_rate": 8.76e-05,
+      "loss": 0.5078,
+      "step": 220
+    },
+    {
+      "epoch": 0.07761966364812418,
+      "grad_norm": 0.3246982991695404,
+      "learning_rate": 9.56e-05,
+      "loss": 0.488,
+      "step": 240
+    },
+    {
+      "epoch": 0.08408796895213454,
+      "grad_norm": 0.341487318277359,
+      "learning_rate": 0.00010360000000000001,
+      "loss": 0.5058,
+      "step": 260
+    },
+    {
+      "epoch": 0.09055627425614489,
+      "grad_norm": 0.3597952127456665,
+      "learning_rate": 0.00011160000000000002,
+      "loss": 0.4574,
+      "step": 280
+    },
+    {
+      "epoch": 0.09702457956015524,
+      "grad_norm": 0.40912091732025146,
+      "learning_rate": 0.00011960000000000001,
+      "loss": 0.4931,
+      "step": 300
+    },
+    {
+      "epoch": 0.1034928848641656,
+      "grad_norm": 0.37430697679519653,
+      "learning_rate": 0.0001276,
+      "loss": 0.4724,
+      "step": 320
+    },
+    {
+      "epoch": 0.10996119016817593,
+      "grad_norm": 0.3920972943305969,
+      "learning_rate": 0.00013560000000000002,
+      "loss": 0.4997,
+      "step": 340
+    },
+    {
+      "epoch": 0.11642949547218628,
+      "grad_norm": 0.32511982321739197,
+      "learning_rate": 0.0001436,
+      "loss": 0.4889,
+      "step": 360
+    },
+    {
+      "epoch": 0.12289780077619664,
+      "grad_norm": 0.3926583528518677,
+      "learning_rate": 0.0001516,
+      "loss": 0.4772,
+      "step": 380
+    },
+    {
+      "epoch": 0.129366106080207,
+      "grad_norm": 0.3535909950733185,
+      "learning_rate": 0.0001596,
+      "loss": 0.465,
+      "step": 400
+    },
+    {
+      "epoch": 0.13583441138421734,
+      "grad_norm": 0.34381625056266785,
+      "learning_rate": 0.0001676,
+      "loss": 0.4934,
+      "step": 420
+    },
+    {
+      "epoch": 0.1423027166882277,
+      "grad_norm": 0.3753703236579895,
+      "learning_rate": 0.0001756,
+      "loss": 0.5068,
+      "step": 440
+    },
+    {
+      "epoch": 0.14877102199223805,
+      "grad_norm": 0.34895479679107666,
+      "learning_rate": 0.00018360000000000002,
+      "loss": 0.5186,
+      "step": 460
+    },
+    {
+      "epoch": 0.15523932729624837,
+      "grad_norm": 0.24720332026481628,
+      "learning_rate": 0.0001916,
+      "loss": 0.461,
+      "step": 480
+    },
+    {
+      "epoch": 0.16170763260025872,
+      "grad_norm": 0.3713696002960205,
+      "learning_rate": 0.0001996,
+      "loss": 0.4932,
+      "step": 500
+    },
+    {
+      "epoch": 0.16817593790426907,
+      "grad_norm": 0.3724994659423828,
+      "learning_rate": 0.00019933145672061927,
+      "loss": 0.5036,
+      "step": 520
+    },
+    {
+      "epoch": 0.17464424320827943,
+      "grad_norm": 0.3562956750392914,
+      "learning_rate": 0.00019862772695285012,
+      "loss": 0.4871,
+      "step": 540
+    },
+    {
+      "epoch": 0.18111254851228978,
+      "grad_norm": 0.331737756729126,
+      "learning_rate": 0.00019792399718508095,
+      "loss": 0.5178,
+      "step": 560
+    },
+    {
+      "epoch": 0.18758085381630013,
+      "grad_norm": 0.34049859642982483,
+      "learning_rate": 0.00019722026741731178,
+      "loss": 0.488,
+      "step": 580
+    },
+    {
+      "epoch": 0.19404915912031048,
+      "grad_norm": 0.36076661944389343,
+      "learning_rate": 0.00019651653764954258,
+      "loss": 0.4821,
+      "step": 600
+    },
+    {
+      "epoch": 0.20051746442432083,
+      "grad_norm": 0.3059399127960205,
+      "learning_rate": 0.0001958128078817734,
+      "loss": 0.4924,
+      "step": 620
+    },
+    {
+      "epoch": 0.2069857697283312,
+      "grad_norm": 0.30692920088768005,
+      "learning_rate": 0.00019510907811400423,
+      "loss": 0.4822,
+      "step": 640
+    },
+    {
+      "epoch": 0.21345407503234154,
+      "grad_norm": 0.3319448232650757,
+      "learning_rate": 0.00019440534834623506,
+      "loss": 0.4607,
+      "step": 660
+    },
+    {
+      "epoch": 0.21992238033635186,
+      "grad_norm": 0.3047769069671631,
+      "learning_rate": 0.00019370161857846588,
+      "loss": 0.5357,
+      "step": 680
+    },
+    {
+      "epoch": 0.22639068564036222,
+      "grad_norm": 0.4353258013725281,
+      "learning_rate": 0.00019299788881069668,
+      "loss": 0.4738,
+      "step": 700
+    },
+    {
+      "epoch": 0.23285899094437257,
+      "grad_norm": 0.30877336859703064,
+      "learning_rate": 0.00019229415904292754,
+      "loss": 0.5128,
+      "step": 720
+    },
+    {
+      "epoch": 0.23932729624838292,
+      "grad_norm": 0.3188530206680298,
+      "learning_rate": 0.00019159042927515836,
+      "loss": 0.4651,
+      "step": 740
+    },
+    {
+      "epoch": 0.24579560155239327,
+      "grad_norm": 0.40678778290748596,
+      "learning_rate": 0.0001908866995073892,
+      "loss": 0.5112,
+      "step": 760
+    },
+    {
+      "epoch": 0.2522639068564036,
+      "grad_norm": 0.34188327193260193,
+      "learning_rate": 0.00019018296973962,
+      "loss": 0.487,
+      "step": 780
+    },
+    {
+      "epoch": 0.258732212160414,
+      "grad_norm": 0.3002854585647583,
+      "learning_rate": 0.00018947923997185081,
+      "loss": 0.512,
+      "step": 800
+    },
+    {
+      "epoch": 0.2652005174644243,
+      "grad_norm": 0.35756272077560425,
+      "learning_rate": 0.00018877551020408164,
+      "loss": 0.4812,
+      "step": 820
+    },
+    {
+      "epoch": 0.2716688227684347,
+      "grad_norm": 0.33527258038520813,
+      "learning_rate": 0.00018807178043631247,
+      "loss": 0.4895,
+      "step": 840
+    },
+    {
+      "epoch": 0.278137128072445,
+      "grad_norm": 0.3472883105278015,
+      "learning_rate": 0.0001873680506685433,
+      "loss": 0.4671,
+      "step": 860
+    },
+    {
+      "epoch": 0.2846054333764554,
+      "grad_norm": 0.3604671061038971,
+      "learning_rate": 0.0001866643209007741,
+      "loss": 0.5233,
+      "step": 880
+    },
+    {
+      "epoch": 0.2910737386804657,
+      "grad_norm": 0.31021153926849365,
+      "learning_rate": 0.00018596059113300492,
+      "loss": 0.491,
+      "step": 900
+    },
+    {
+      "epoch": 0.2975420439844761,
+      "grad_norm": 0.6417067050933838,
+      "learning_rate": 0.00018525686136523577,
+      "loss": 0.4858,
+      "step": 920
+    },
+    {
+      "epoch": 0.3040103492884864,
+      "grad_norm": 0.33719781041145325,
+      "learning_rate": 0.0001845531315974666,
+      "loss": 0.4722,
+      "step": 940
+    },
+    {
+      "epoch": 0.31047865459249674,
+      "grad_norm": 0.3568169176578522,
+      "learning_rate": 0.0001838494018296974,
+      "loss": 0.5088,
+      "step": 960
+    },
+    {
+      "epoch": 0.3169469598965071,
+      "grad_norm": 0.42162615060806274,
+      "learning_rate": 0.00018314567206192823,
+      "loss": 0.5091,
+      "step": 980
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "grad_norm": 0.3212260603904724,
+      "learning_rate": 0.00018244194229415905,
+      "loss": 0.4654,
+      "step": 1000
+    },
+    {
+      "epoch": 0.32341526520051744,
+      "eval_loss": 0.6167479753494263,
+      "eval_runtime": 1013.3659,
+      "eval_samples_per_second": 6.046,
+      "eval_steps_per_second": 0.756,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3298835705045278,
+      "grad_norm": 0.3184376060962677,
+      "learning_rate": 0.00018173821252638988,
+      "loss": 0.4712,
+      "step": 1020
+    },
+    {
+      "epoch": 0.33635187580853815,
+      "grad_norm": 0.3114645183086395,
+      "learning_rate": 0.0001810344827586207,
+      "loss": 0.4393,
+      "step": 1040
+    },
+    {
+      "epoch": 0.3428201811125485,
+      "grad_norm": 0.30122488737106323,
+      "learning_rate": 0.0001803307529908515,
+      "loss": 0.4882,
+      "step": 1060
+    },
+    {
+      "epoch": 0.34928848641655885,
+      "grad_norm": 0.2877393066883087,
+      "learning_rate": 0.00017962702322308233,
+      "loss": 0.413,
+      "step": 1080
+    },
+    {
+      "epoch": 0.35575679172056923,
+      "grad_norm": 0.33227887749671936,
+      "learning_rate": 0.00017892329345531319,
+      "loss": 0.4527,
+      "step": 1100
+    },
+    {
+      "epoch": 0.36222509702457956,
+      "grad_norm": 0.33423128724098206,
+      "learning_rate": 0.000178219563687544,
+      "loss": 0.4604,
+      "step": 1120
+    },
+    {
+      "epoch": 0.36869340232858994,
+      "grad_norm": 0.36922487616539,
+      "learning_rate": 0.0001775158339197748,
+      "loss": 0.5044,
+      "step": 1140
+    },
+    {
+      "epoch": 0.37516170763260026,
+      "grad_norm": 0.34816262125968933,
+      "learning_rate": 0.00017681210415200564,
+      "loss": 0.4898,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3816300129366106,
+      "grad_norm": 0.32086044549942017,
+      "learning_rate": 0.00017610837438423646,
+      "loss": 0.4647,
+      "step": 1180
+    },
+    {
+      "epoch": 0.38809831824062097,
+      "grad_norm": 0.3879236876964569,
+      "learning_rate": 0.0001754046446164673,
+      "loss": 0.5179,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3945666235446313,
+      "grad_norm": 0.2899722158908844,
+      "learning_rate": 0.00017470091484869812,
+      "loss": 0.4786,
+      "step": 1220
+    },
+    {
+      "epoch": 0.40103492884864167,
+      "grad_norm": 0.3844458758831024,
+      "learning_rate": 0.00017399718508092892,
+      "loss": 0.4513,
+      "step": 1240
+    },
+    {
+      "epoch": 0.407503234152652,
+      "grad_norm": 0.3678169548511505,
+      "learning_rate": 0.00017329345531315974,
+      "loss": 0.4789,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4139715394566624,
+      "grad_norm": 0.37138116359710693,
+      "learning_rate": 0.00017258972554539057,
+      "loss": 0.4694,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4204398447606727,
+      "grad_norm": 0.3604221045970917,
+      "learning_rate": 0.00017188599577762142,
+      "loss": 0.4496,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4269081500646831,
+      "grad_norm": 0.33961811661720276,
+      "learning_rate": 0.00017118226600985222,
+      "loss": 0.4696,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4333764553686934,
+      "grad_norm": 0.3244430124759674,
+      "learning_rate": 0.00017047853624208305,
+      "loss": 0.4322,
+      "step": 1340
+    },
+    {
+      "epoch": 0.4398447606727037,
+      "grad_norm": 0.2879267930984497,
+      "learning_rate": 0.00016977480647431388,
+      "loss": 0.4646,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4463130659767141,
+      "grad_norm": 0.3437345027923584,
+      "learning_rate": 0.0001690710767065447,
+      "loss": 0.4711,
+      "step": 1380
+    },
+    {
+      "epoch": 0.45278137128072443,
+      "grad_norm": 0.3166882395744324,
+      "learning_rate": 0.00016836734693877553,
+      "loss": 0.486,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4592496765847348,
+      "grad_norm": 0.3677555322647095,
+      "learning_rate": 0.00016766361717100633,
+      "loss": 0.4677,
+      "step": 1420
+    },
+    {
+      "epoch": 0.46571798188874514,
+      "grad_norm": 0.32657769322395325,
+      "learning_rate": 0.00016695988740323715,
+      "loss": 0.4649,
+      "step": 1440
+    },
+    {
+      "epoch": 0.4721862871927555,
+      "grad_norm": 0.2980731427669525,
+      "learning_rate": 0.00016625615763546798,
+      "loss": 0.4547,
+      "step": 1460
+    },
+    {
+      "epoch": 0.47865459249676584,
+      "grad_norm": 0.3690549433231354,
+      "learning_rate": 0.0001655524278676988,
+      "loss": 0.4512,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4851228978007762,
+      "grad_norm": 0.31456947326660156,
+      "learning_rate": 0.00016484869809992963,
+      "loss": 0.4675,
+      "step": 1500
+    },
+    {
+      "epoch": 0.49159120310478654,
+      "grad_norm": 0.3823802173137665,
+      "learning_rate": 0.00016414496833216046,
+      "loss": 0.4849,
+      "step": 1520
+    },
+    {
+      "epoch": 0.49805950840879687,
+      "grad_norm": 0.33280324935913086,
+      "learning_rate": 0.0001634412385643913,
+      "loss": 0.4876,
+      "step": 1540
+    },
+    {
+      "epoch": 0.5045278137128072,
+      "grad_norm": 0.28788062930107117,
+      "learning_rate": 0.00016273750879662211,
+      "loss": 0.426,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5109961190168176,
+      "grad_norm": 0.3809375762939453,
+      "learning_rate": 0.00016203377902885294,
+      "loss": 0.4676,
+      "step": 1580
+    },
+    {
+      "epoch": 0.517464424320828,
+      "grad_norm": 0.27940312027931213,
+      "learning_rate": 0.00016133004926108374,
+      "loss": 0.4413,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5239327296248383,
+      "grad_norm": 0.30946919322013855,
+      "learning_rate": 0.00016062631949331457,
+      "loss": 0.4395,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5304010349288486,
+      "grad_norm": 0.36931583285331726,
+      "learning_rate": 0.0001599225897255454,
+      "loss": 0.4565,
+      "step": 1640
+    },
+    {
+      "epoch": 0.536869340232859,
+      "grad_norm": 0.3652470111846924,
+      "learning_rate": 0.00015921885995777622,
+      "loss": 0.4522,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5433376455368694,
+      "grad_norm": 0.3060782551765442,
+      "learning_rate": 0.00015851513019000705,
+      "loss": 0.45,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5498059508408797,
+      "grad_norm": 0.3275803029537201,
+      "learning_rate": 0.00015781140042223787,
+      "loss": 0.445,
+      "step": 1700
+    },
+    {
+      "epoch": 0.55627425614489,
+      "grad_norm": 0.34865352511405945,
+      "learning_rate": 0.0001571076706544687,
+      "loss": 0.4306,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5627425614489003,
+      "grad_norm": 0.42525383830070496,
+      "learning_rate": 0.00015640394088669953,
+      "loss": 0.4544,
+      "step": 1740
+    },
+    {
+      "epoch": 0.5692108667529108,
+      "grad_norm": 0.32165050506591797,
+      "learning_rate": 0.00015570021111893035,
+      "loss": 0.484,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5756791720569211,
+      "grad_norm": 0.3213440477848053,
+      "learning_rate": 0.00015499648135116115,
+      "loss": 0.4469,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5821474773609314,
+      "grad_norm": 0.42835432291030884,
+      "learning_rate": 0.00015429275158339198,
+      "loss": 0.4592,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5886157826649417,
+      "grad_norm": 0.3861497938632965,
+      "learning_rate": 0.0001535890218156228,
+      "loss": 0.4607,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5950840879689522,
+      "grad_norm": 0.4156148135662079,
+      "learning_rate": 0.00015288529204785363,
+      "loss": 0.4541,
+      "step": 1840
+    },
+    {
+      "epoch": 0.6015523932729625,
+      "grad_norm": 0.32371824979782104,
+      "learning_rate": 0.00015218156228008446,
+      "loss": 0.4287,
+      "step": 1860
+    },
+    {
+      "epoch": 0.6080206985769728,
+      "grad_norm": 0.363788366317749,
+      "learning_rate": 0.00015147783251231528,
+      "loss": 0.428,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6144890038809832,
+      "grad_norm": 0.3932839334011078,
+      "learning_rate": 0.0001507741027445461,
+      "loss": 0.4359,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6209573091849935,
+      "grad_norm": 0.3009307086467743,
+      "learning_rate": 0.00015007037297677694,
+      "loss": 0.3976,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6274256144890039,
+      "grad_norm": 0.3484659194946289,
+      "learning_rate": 0.00014936664320900776,
+      "loss": 0.4584,
+      "step": 1940
+    },
+    {
+      "epoch": 0.6338939197930142,
+      "grad_norm": 0.33728504180908203,
+      "learning_rate": 0.00014866291344123856,
+      "loss": 0.4543,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6403622250970246,
+      "grad_norm": 0.3207024037837982,
+      "learning_rate": 0.0001479591836734694,
+      "loss": 0.4611,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "grad_norm": 0.29134076833724976,
+      "learning_rate": 0.00014725545390570022,
+      "loss": 0.439,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6468305304010349,
+      "eval_loss": 0.6003134250640869,
+      "eval_runtime": 968.0302,
+      "eval_samples_per_second": 6.329,
+      "eval_steps_per_second": 0.791,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6532988357050453,
+      "grad_norm": 0.3548874258995056,
+      "learning_rate": 0.00014655172413793104,
+      "loss": 0.4548,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6597671410090556,
+      "grad_norm": 0.3707274794578552,
+      "learning_rate": 0.00014584799437016187,
+      "loss": 0.4271,
+      "step": 2040
+    },
+    {
+      "epoch": 0.666235446313066,
+      "grad_norm": 0.37172695994377136,
+      "learning_rate": 0.0001451442646023927,
+      "loss": 0.4626,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6727037516170763,
+      "grad_norm": 0.3459303081035614,
+      "learning_rate": 0.00014444053483462352,
+      "loss": 0.4317,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6791720569210866,
+      "grad_norm": 0.3477213680744171,
+      "learning_rate": 0.00014373680506685435,
+      "loss": 0.4243,
+      "step": 2100
+    },
+    {
+      "epoch": 0.685640362225097,
+      "grad_norm": 0.31847599148750305,
+      "learning_rate": 0.00014303307529908517,
+      "loss": 0.4234,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6921086675291074,
+      "grad_norm": 0.3231470584869385,
+      "learning_rate": 0.00014232934553131597,
+      "loss": 0.4302,
+      "step": 2140
+    },
+    {
+      "epoch": 0.6985769728331177,
+      "grad_norm": 0.3745759427547455,
+      "learning_rate": 0.0001416256157635468,
+      "loss": 0.4533,
+      "step": 2160
+    },
+    {
+      "epoch": 0.705045278137128,
+      "grad_norm": 0.34834805130958557,
+      "learning_rate": 0.00014092188599577763,
+      "loss": 0.4748,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7115135834411385,
+      "grad_norm": 0.3675827085971832,
+      "learning_rate": 0.00014021815622800845,
+      "loss": 0.4401,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7179818887451488,
+      "grad_norm": 0.2839801013469696,
+      "learning_rate": 0.00013951442646023928,
+      "loss": 0.4332,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7244501940491591,
+      "grad_norm": 0.27037495374679565,
+      "learning_rate": 0.00013881069669247008,
+      "loss": 0.4382,
+      "step": 2240
+    },
+    {
+      "epoch": 0.7309184993531694,
+      "grad_norm": 0.34917208552360535,
+      "learning_rate": 0.00013810696692470093,
+      "loss": 0.4251,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7373868046571799,
+      "grad_norm": 0.39801281690597534,
+      "learning_rate": 0.00013740323715693176,
+      "loss": 0.4432,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7438551099611902,
+      "grad_norm": 0.8932256102561951,
+      "learning_rate": 0.00013669950738916259,
+      "loss": 0.4577,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7503234152652005,
+      "grad_norm": 0.35125109553337097,
+      "learning_rate": 0.00013599577762139339,
+      "loss": 0.4502,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7567917205692108,
+      "grad_norm": 0.3012046813964844,
+      "learning_rate": 0.0001352920478536242,
+      "loss": 0.4513,
+      "step": 2340
+    },
+    {
+      "epoch": 0.7632600258732212,
+      "grad_norm": 0.3242063820362091,
+      "learning_rate": 0.00013458831808585504,
+      "loss": 0.4559,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7697283311772316,
+      "grad_norm": 0.3523860275745392,
+      "learning_rate": 0.00013388458831808587,
+      "loss": 0.4434,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7761966364812419,
+      "grad_norm": 0.29160013794898987,
+      "learning_rate": 0.0001331808585503167,
+      "loss": 0.44,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7826649417852523,
+      "grad_norm": 0.3047971725463867,
+      "learning_rate": 0.0001324771287825475,
+      "loss": 0.4336,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7891332470892626,
+      "grad_norm": 0.3192126750946045,
+      "learning_rate": 0.00013177339901477832,
+      "loss": 0.4495,
+      "step": 2440
+    },
+    {
+      "epoch": 0.795601552393273,
+      "grad_norm": 0.37997299432754517,
+      "learning_rate": 0.00013106966924700917,
+      "loss": 0.4685,
+      "step": 2460
+    },
+    {
+      "epoch": 0.8020698576972833,
+      "grad_norm": 0.2811839282512665,
+      "learning_rate": 0.00013036593947924,
+      "loss": 0.4206,
+      "step": 2480
+    },
+    {
+      "epoch": 0.8085381630012937,
+      "grad_norm": 0.3121369779109955,
+      "learning_rate": 0.0001296622097114708,
+      "loss": 0.425,
+      "step": 2500
+    },
+    {
+      "epoch": 0.815006468305304,
+      "grad_norm": 0.3134746253490448,
+      "learning_rate": 0.00012895847994370162,
+      "loss": 0.4498,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8214747736093143,
+      "grad_norm": 0.36752375960350037,
+      "learning_rate": 0.00012825475017593245,
+      "loss": 0.4288,
+      "step": 2540
+    },
+    {
+      "epoch": 0.8279430789133247,
+      "grad_norm": 0.3915141224861145,
+      "learning_rate": 0.00012755102040816328,
+      "loss": 0.4605,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8344113842173351,
+      "grad_norm": 0.3662431538105011,
+      "learning_rate": 0.00012684729064039408,
+      "loss": 0.4194,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8408796895213454,
+      "grad_norm": 0.3387574851512909,
+      "learning_rate": 0.0001261435608726249,
+      "loss": 0.4258,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8473479948253557,
+      "grad_norm": 0.38025060296058655,
+      "learning_rate": 0.00012543983110485573,
+      "loss": 0.4291,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8538163001293662,
+      "grad_norm": 0.3517441749572754,
+      "learning_rate": 0.00012473610133708658,
+      "loss": 0.4384,
+      "step": 2640
+    },
+    {
+      "epoch": 0.8602846054333765,
+      "grad_norm": 0.33037152886390686,
+      "learning_rate": 0.0001240323715693174,
+      "loss": 0.4452,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8667529107373868,
+      "grad_norm": 0.3645654618740082,
+      "learning_rate": 0.0001233286418015482,
+      "loss": 0.3992,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8732212160413971,
+      "grad_norm": 0.32396814227104187,
+      "learning_rate": 0.00012262491203377904,
+      "loss": 0.4,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8796895213454075,
+      "grad_norm": 0.3377726078033447,
+      "learning_rate": 0.00012192118226600986,
+      "loss": 0.4306,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8861578266494179,
+      "grad_norm": 0.32345983386039734,
+      "learning_rate": 0.00012121745249824067,
+      "loss": 0.4225,
+      "step": 2740
+    },
+    {
+      "epoch": 0.8926261319534282,
+      "grad_norm": 0.34778884053230286,
+      "learning_rate": 0.0001205137227304715,
+      "loss": 0.4256,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8990944372574385,
+      "grad_norm": 0.3861066997051239,
+      "learning_rate": 0.00011980999296270233,
+      "loss": 0.4465,
+      "step": 2780
+    },
+    {
+      "epoch": 0.9055627425614489,
+      "grad_norm": 0.3882784843444824,
+      "learning_rate": 0.00011910626319493314,
+      "loss": 0.4026,
+      "step": 2800
+    },
+    {
+      "epoch": 0.9120310478654593,
+      "grad_norm": 0.4380407929420471,
+      "learning_rate": 0.00011840253342716397,
+      "loss": 0.4375,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9184993531694696,
+      "grad_norm": 0.35943394899368286,
+      "learning_rate": 0.00011769880365939481,
+      "loss": 0.4337,
+      "step": 2840
+    },
+    {
+      "epoch": 0.92496765847348,
+      "grad_norm": 0.3862978219985962,
+      "learning_rate": 0.00011699507389162563,
+      "loss": 0.3949,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9314359637774903,
+      "grad_norm": 0.34592369198799133,
+      "learning_rate": 0.00011629134412385645,
+      "loss": 0.3798,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9379042690815006,
+      "grad_norm": 0.34942877292633057,
+      "learning_rate": 0.00011558761435608727,
+      "loss": 0.4235,
+      "step": 2900
+    },
+    {
+      "epoch": 0.944372574385511,
+      "grad_norm": 0.28018903732299805,
+      "learning_rate": 0.00011488388458831809,
+      "loss": 0.4353,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9508408796895214,
+      "grad_norm": 0.3505672812461853,
+      "learning_rate": 0.00011418015482054891,
+      "loss": 0.3954,
+      "step": 2940
+    },
+    {
+      "epoch": 0.9573091849935317,
+      "grad_norm": 0.3264220654964447,
+      "learning_rate": 0.00011347642505277974,
+      "loss": 0.4422,
+      "step": 2960
+    },
+    {
+      "epoch": 0.963777490297542,
+      "grad_norm": 0.36382535099983215,
+      "learning_rate": 0.00011277269528501055,
+      "loss": 0.4116,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "grad_norm": 0.3051494359970093,
+      "learning_rate": 0.00011206896551724138,
+      "loss": 0.4135,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9702457956015524,
+      "eval_loss": 0.5850913524627686,
+      "eval_runtime": 967.6256,
+      "eval_samples_per_second": 6.332,
+      "eval_steps_per_second": 0.792,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9767141009055628,
+      "grad_norm": 0.34533271193504333,
+      "learning_rate": 0.00011136523574947222,
+      "loss": 0.4312,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9831824062095731,
+      "grad_norm": 0.35221609473228455,
+      "learning_rate": 0.00011066150598170305,
+      "loss": 0.4245,
+      "step": 3040
+    },
+    {
+      "epoch": 0.9896507115135834,
+      "grad_norm": 0.3893713057041168,
+      "learning_rate": 0.00010995777621393386,
+      "loss": 0.4048,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9961190168175937,
+      "grad_norm": 0.3586278557777405,
+      "learning_rate": 0.00010925404644616468,
+      "loss": 0.4159,
+      "step": 3080
+    },
+    {
+      "epoch": 1.0025873221216042,
+      "grad_norm": 0.32620444893836975,
+      "learning_rate": 0.0001085503166783955,
+      "loss": 0.3741,
+      "step": 3100
+    },
+    {
+      "epoch": 1.0090556274256144,
+      "grad_norm": 0.33812645077705383,
+      "learning_rate": 0.00010784658691062632,
+      "loss": 0.3694,
+      "step": 3120
+    },
+    {
+      "epoch": 1.0155239327296248,
+      "grad_norm": 0.39038708806037903,
+      "learning_rate": 0.00010714285714285715,
+      "loss": 0.3882,
+      "step": 3140
+    },
+    {
+      "epoch": 1.0219922380336353,
+      "grad_norm": 0.3209969997406006,
+      "learning_rate": 0.00010643912737508796,
+      "loss": 0.3437,
+      "step": 3160
+    },
+    {
+      "epoch": 1.0284605433376455,
+      "grad_norm": 0.3705078661441803,
+      "learning_rate": 0.00010573539760731879,
+      "loss": 0.3568,
+      "step": 3180
+    },
+    {
+      "epoch": 1.034928848641656,
+      "grad_norm": 0.3562203645706177,
+      "learning_rate": 0.0001050316678395496,
+      "loss": 0.3916,
+      "step": 3200
+    },
+    {
+      "epoch": 1.0413971539456663,
+      "grad_norm": 0.3724074363708496,
+      "learning_rate": 0.00010432793807178046,
+      "loss": 0.3866,
+      "step": 3220
+    },
+    {
+      "epoch": 1.0478654592496766,
+      "grad_norm": 0.40453559160232544,
+      "learning_rate": 0.00010362420830401127,
+      "loss": 0.368,
+      "step": 3240
+    },
+    {
+      "epoch": 1.054333764553687,
+      "grad_norm": 0.3316144049167633,
+      "learning_rate": 0.0001029204785362421,
+      "loss": 0.3696,
+      "step": 3260
+    },
+    {
+      "epoch": 1.0608020698576972,
+      "grad_norm": 0.3503650724887848,
+      "learning_rate": 0.00010221674876847291,
+      "loss": 0.3644,
+      "step": 3280
+    },
+    {
+      "epoch": 1.0672703751617076,
+      "grad_norm": 0.3136923909187317,
+      "learning_rate": 0.00010151301900070374,
+      "loss": 0.3414,
+      "step": 3300
+    },
+    {
+      "epoch": 1.073738680465718,
+      "grad_norm": 0.3573864996433258,
+      "learning_rate": 0.00010080928923293456,
+      "loss": 0.3586,
+      "step": 3320
+    },
+    {
+      "epoch": 1.0802069857697283,
+      "grad_norm": 0.26215964555740356,
+      "learning_rate": 0.00010010555946516538,
+      "loss": 0.356,
+      "step": 3340
+    },
+    {
+      "epoch": 1.0866752910737387,
+      "grad_norm": 0.3103466331958771,
+      "learning_rate": 9.940182969739622e-05,
+      "loss": 0.3478,
+      "step": 3360
+    },
+    {
+      "epoch": 1.093143596377749,
+      "grad_norm": 0.3601842224597931,
+      "learning_rate": 9.869809992962703e-05,
+      "loss": 0.3635,
+      "step": 3380
+    },
+    {
+      "epoch": 1.0996119016817594,
+      "grad_norm": 0.3715918958187103,
+      "learning_rate": 9.799437016185785e-05,
+      "loss": 0.3489,
+      "step": 3400
+    },
+    {
+      "epoch": 1.1060802069857698,
+      "grad_norm": 0.4216213524341583,
+      "learning_rate": 9.729064039408867e-05,
+      "loss": 0.3461,
+      "step": 3420
+    },
+    {
+      "epoch": 1.11254851228978,
+      "grad_norm": 0.3878704905509949,
+      "learning_rate": 9.65869106263195e-05,
+      "loss": 0.3548,
+      "step": 3440
+    },
+    {
+      "epoch": 1.1190168175937905,
+      "grad_norm": 0.36658766865730286,
+      "learning_rate": 9.588318085855032e-05,
+      "loss": 0.3395,
+      "step": 3460
+    },
+    {
+      "epoch": 1.1254851228978007,
+      "grad_norm": 0.38134855031967163,
+      "learning_rate": 9.517945109078115e-05,
+      "loss": 0.3829,
+      "step": 3480
+    },
+    {
+      "epoch": 1.131953428201811,
+      "grad_norm": 0.42683905363082886,
+      "learning_rate": 9.447572132301197e-05,
+      "loss": 0.3914,
+      "step": 3500
+    },
+    {
+      "epoch": 1.1384217335058215,
+      "grad_norm": 0.3340086042881012,
+      "learning_rate": 9.377199155524279e-05,
+      "loss": 0.3301,
+      "step": 3520
+    },
+    {
+      "epoch": 1.1448900388098318,
+      "grad_norm": 0.37897104024887085,
+      "learning_rate": 9.306826178747361e-05,
+      "loss": 0.3536,
+      "step": 3540
+    },
+    {
+      "epoch": 1.1513583441138422,
+      "grad_norm": 0.361061692237854,
+      "learning_rate": 9.236453201970444e-05,
+      "loss": 0.338,
+      "step": 3560
+    },
+    {
+      "epoch": 1.1578266494178524,
+      "grad_norm": 0.34730997681617737,
+      "learning_rate": 9.166080225193527e-05,
+      "loss": 0.3679,
+      "step": 3580
+    },
+    {
+      "epoch": 1.1642949547218628,
+      "grad_norm": 0.29455122351646423,
+      "learning_rate": 9.095707248416608e-05,
+      "loss": 0.3602,
+      "step": 3600
+    },
+    {
+      "epoch": 1.1707632600258733,
+      "grad_norm": 0.33159056305885315,
+      "learning_rate": 9.02533427163969e-05,
+      "loss": 0.3359,
+      "step": 3620
+    },
+    {
+      "epoch": 1.1772315653298835,
+      "grad_norm": 0.48726096749305725,
+      "learning_rate": 8.954961294862773e-05,
+      "loss": 0.3879,
+      "step": 3640
+    },
+    {
+      "epoch": 1.183699870633894,
+      "grad_norm": 0.3182481825351715,
+      "learning_rate": 8.884588318085856e-05,
+      "loss": 0.3392,
+      "step": 3660
+    },
+    {
+      "epoch": 1.1901681759379044,
+      "grad_norm": 0.3488766551017761,
+      "learning_rate": 8.814215341308939e-05,
+      "loss": 0.3419,
+      "step": 3680
+    },
+    {
+      "epoch": 1.1966364812419146,
+      "grad_norm": 0.34345540404319763,
+      "learning_rate": 8.74384236453202e-05,
+      "loss": 0.3598,
+      "step": 3700
+    },
+    {
+      "epoch": 1.203104786545925,
+      "grad_norm": 0.37642374634742737,
+      "learning_rate": 8.673469387755102e-05,
+      "loss": 0.3578,
+      "step": 3720
+    },
+    {
+      "epoch": 1.2095730918499352,
+      "grad_norm": 0.3726005554199219,
+      "learning_rate": 8.603096410978185e-05,
+      "loss": 0.3547,
+      "step": 3740
+    },
+    {
+      "epoch": 1.2160413971539457,
+      "grad_norm": 0.36603832244873047,
+      "learning_rate": 8.532723434201268e-05,
+      "loss": 0.3831,
+      "step": 3760
+    },
+    {
+      "epoch": 1.222509702457956,
+      "grad_norm": 0.345103919506073,
+      "learning_rate": 8.462350457424349e-05,
+      "loss": 0.3563,
+      "step": 3780
+    },
+    {
+      "epoch": 1.2289780077619663,
+      "grad_norm": 0.31071504950523376,
+      "learning_rate": 8.391977480647432e-05,
+      "loss": 0.3263,
+      "step": 3800
+    },
+    {
+      "epoch": 1.2354463130659767,
+      "grad_norm": 0.3534263074398041,
+      "learning_rate": 8.321604503870513e-05,
+      "loss": 0.3591,
+      "step": 3820
+    },
+    {
+      "epoch": 1.2419146183699872,
+      "grad_norm": 0.3568720817565918,
+      "learning_rate": 8.251231527093597e-05,
+      "loss": 0.3815,
+      "step": 3840
+    },
+    {
+      "epoch": 1.2483829236739974,
+      "grad_norm": 0.3450963795185089,
+      "learning_rate": 8.18085855031668e-05,
+      "loss": 0.3481,
+      "step": 3860
+    },
+    {
+      "epoch": 1.2548512289780078,
+      "grad_norm": 0.3554648160934448,
+      "learning_rate": 8.110485573539761e-05,
+      "loss": 0.3364,
+      "step": 3880
+    },
+    {
+      "epoch": 1.261319534282018,
+      "grad_norm": 0.4188801348209381,
+      "learning_rate": 8.040112596762844e-05,
+      "loss": 0.3686,
+      "step": 3900
+    },
+    {
+      "epoch": 1.2677878395860285,
+      "grad_norm": 0.3129253089427948,
+      "learning_rate": 7.969739619985925e-05,
+      "loss": 0.3664,
+      "step": 3920
+    },
+    {
+      "epoch": 1.274256144890039,
+      "grad_norm": 0.3885388970375061,
+      "learning_rate": 7.899366643209009e-05,
+      "loss": 0.3415,
+      "step": 3940
+    },
+    {
+      "epoch": 1.2807244501940491,
+      "grad_norm": 0.4131453335285187,
+      "learning_rate": 7.82899366643209e-05,
+      "loss": 0.3552,
+      "step": 3960
+    },
+    {
+      "epoch": 1.2871927554980596,
+      "grad_norm": 0.39486396312713623,
+      "learning_rate": 7.758620689655173e-05,
+      "loss": 0.366,
+      "step": 3980
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "grad_norm": 0.3580794930458069,
+      "learning_rate": 7.688247712878254e-05,
+      "loss": 0.3585,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2936610608020698,
+      "eval_loss": 0.5952740907669067,
+      "eval_runtime": 967.8131,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 4000
+    },
+    {
+      "epoch": 1.3001293661060802,
+      "grad_norm": 0.3903474509716034,
+      "learning_rate": 7.617874736101337e-05,
+      "loss": 0.3347,
+      "step": 4020
+    },
+    {
+      "epoch": 1.3065976714100906,
+      "grad_norm": 0.4154898226261139,
+      "learning_rate": 7.54750175932442e-05,
+      "loss": 0.3314,
+      "step": 4040
+    },
+    {
+      "epoch": 1.3130659767141009,
+      "grad_norm": 0.33763521909713745,
+      "learning_rate": 7.477128782547502e-05,
+      "loss": 0.3552,
+      "step": 4060
+    },
+    {
+      "epoch": 1.3195342820181113,
+      "grad_norm": 0.5618635416030884,
+      "learning_rate": 7.406755805770585e-05,
+      "loss": 0.363,
+      "step": 4080
+    },
+    {
+      "epoch": 1.3260025873221215,
+      "grad_norm": 0.37114137411117554,
+      "learning_rate": 7.336382828993666e-05,
+      "loss": 0.3421,
+      "step": 4100
+    },
+    {
+      "epoch": 1.332470892626132,
+      "grad_norm": 0.34323641657829285,
+      "learning_rate": 7.266009852216749e-05,
+      "loss": 0.3659,
+      "step": 4120
+    },
+    {
+      "epoch": 1.3389391979301424,
+      "grad_norm": 0.3527199625968933,
+      "learning_rate": 7.195636875439831e-05,
+      "loss": 0.3412,
+      "step": 4140
+    },
+    {
+      "epoch": 1.3454075032341526,
+      "grad_norm": 0.3413406014442444,
+      "learning_rate": 7.125263898662914e-05,
+      "loss": 0.338,
+      "step": 4160
+    },
+    {
+      "epoch": 1.351875808538163,
+      "grad_norm": 0.34787651896476746,
+      "learning_rate": 7.054890921885995e-05,
+      "loss": 0.3681,
+      "step": 4180
+    },
+    {
+      "epoch": 1.3583441138421732,
+      "grad_norm": 0.2709845006465912,
+      "learning_rate": 6.984517945109078e-05,
+      "loss": 0.325,
+      "step": 4200
+    },
+    {
+      "epoch": 1.3648124191461837,
+      "grad_norm": 0.30966514348983765,
+      "learning_rate": 6.91414496833216e-05,
+      "loss": 0.3677,
+      "step": 4220
+    },
+    {
+      "epoch": 1.371280724450194,
+      "grad_norm": 0.35980409383773804,
+      "learning_rate": 6.843771991555243e-05,
+      "loss": 0.335,
+      "step": 4240
+    },
+    {
+      "epoch": 1.3777490297542043,
+      "grad_norm": 0.5173650979995728,
+      "learning_rate": 6.773399014778326e-05,
+      "loss": 0.348,
+      "step": 4260
+    },
+    {
+      "epoch": 1.3842173350582148,
+      "grad_norm": 0.3048713207244873,
+      "learning_rate": 6.703026038001407e-05,
+      "loss": 0.3323,
+      "step": 4280
+    },
+    {
+      "epoch": 1.390685640362225,
+      "grad_norm": 0.3427985906600952,
+      "learning_rate": 6.63265306122449e-05,
+      "loss": 0.3571,
+      "step": 4300
+    },
+    {
+      "epoch": 1.3971539456662354,
+      "grad_norm": 0.406012624502182,
+      "learning_rate": 6.562280084447573e-05,
+      "loss": 0.3546,
+      "step": 4320
+    },
+    {
+      "epoch": 1.4036222509702458,
+      "grad_norm": 0.31981325149536133,
+      "learning_rate": 6.491907107670655e-05,
+      "loss": 0.3281,
+      "step": 4340
+    },
+    {
+      "epoch": 1.4100905562742563,
+      "grad_norm": 0.3936704099178314,
+      "learning_rate": 6.421534130893737e-05,
+      "loss": 0.3444,
+      "step": 4360
+    },
+    {
+      "epoch": 1.4165588615782665,
+      "grad_norm": 0.3658936023712158,
+      "learning_rate": 6.351161154116819e-05,
+      "loss": 0.339,
+      "step": 4380
+    },
+    {
+      "epoch": 1.4230271668822767,
+      "grad_norm": 0.32523271441459656,
+      "learning_rate": 6.280788177339902e-05,
+      "loss": 0.3311,
+      "step": 4400
+    },
+    {
+      "epoch": 1.4294954721862871,
+      "grad_norm": 0.4120417833328247,
+      "learning_rate": 6.210415200562984e-05,
+      "loss": 0.3342,
+      "step": 4420
+    },
+    {
+      "epoch": 1.4359637774902976,
+      "grad_norm": 0.35293978452682495,
+      "learning_rate": 6.140042223786067e-05,
+      "loss": 0.3561,
+      "step": 4440
+    },
+    {
+      "epoch": 1.442432082794308,
+      "grad_norm": 0.33231040835380554,
+      "learning_rate": 6.0696692470091484e-05,
+      "loss": 0.3573,
+      "step": 4460
+    },
+    {
+      "epoch": 1.4489003880983182,
+      "grad_norm": 0.4345926344394684,
+      "learning_rate": 5.999296270232231e-05,
+      "loss": 0.3364,
+      "step": 4480
+    },
+    {
+      "epoch": 1.4553686934023287,
+      "grad_norm": 0.34958288073539734,
+      "learning_rate": 5.928923293455313e-05,
+      "loss": 0.3162,
+      "step": 4500
+    },
+    {
+      "epoch": 1.4618369987063389,
+      "grad_norm": 0.4669477045536041,
+      "learning_rate": 5.8585503166783964e-05,
+      "loss": 0.3455,
+      "step": 4520
+    },
+    {
+      "epoch": 1.4683053040103493,
+      "grad_norm": 0.32660993933677673,
+      "learning_rate": 5.788177339901478e-05,
+      "loss": 0.3228,
+      "step": 4540
+    },
+    {
+      "epoch": 1.4747736093143597,
+      "grad_norm": 0.3654051423072815,
+      "learning_rate": 5.71780436312456e-05,
+      "loss": 0.3293,
+      "step": 4560
+    },
+    {
+      "epoch": 1.48124191461837,
+      "grad_norm": 0.3496916592121124,
+      "learning_rate": 5.647431386347642e-05,
+      "loss": 0.3382,
+      "step": 4580
+    },
+    {
+      "epoch": 1.4877102199223804,
+      "grad_norm": 0.3890388309955597,
+      "learning_rate": 5.577058409570724e-05,
+      "loss": 0.3468,
+      "step": 4600
+    },
+    {
+      "epoch": 1.4941785252263906,
+      "grad_norm": 0.3924092650413513,
+      "learning_rate": 5.5066854327938076e-05,
+      "loss": 0.3467,
+      "step": 4620
+    },
+    {
+      "epoch": 1.500646830530401,
+      "grad_norm": 0.4015033543109894,
+      "learning_rate": 5.4363124560168896e-05,
+      "loss": 0.3478,
+      "step": 4640
+    },
+    {
+      "epoch": 1.5071151358344115,
+      "grad_norm": 0.34742093086242676,
+      "learning_rate": 5.365939479239972e-05,
+      "loss": 0.3252,
+      "step": 4660
+    },
+    {
+      "epoch": 1.5135834411384217,
+      "grad_norm": 0.3477989435195923,
+      "learning_rate": 5.295566502463054e-05,
+      "loss": 0.3627,
+      "step": 4680
+    },
+    {
+      "epoch": 1.5200517464424321,
+      "grad_norm": 0.4194832146167755,
+      "learning_rate": 5.2251935256861375e-05,
+      "loss": 0.3403,
+      "step": 4700
+    },
+    {
+      "epoch": 1.5265200517464423,
+      "grad_norm": 0.37723594903945923,
+      "learning_rate": 5.1548205489092195e-05,
+      "loss": 0.3424,
+      "step": 4720
+    },
+    {
+      "epoch": 1.5329883570504528,
+      "grad_norm": 0.3695043921470642,
+      "learning_rate": 5.0844475721323015e-05,
+      "loss": 0.3339,
+      "step": 4740
+    },
+    {
+      "epoch": 1.5394566623544632,
+      "grad_norm": 0.35015031695365906,
+      "learning_rate": 5.0140745953553834e-05,
+      "loss": 0.3262,
+      "step": 4760
+    },
+    {
+      "epoch": 1.5459249676584734,
+      "grad_norm": 0.39946597814559937,
+      "learning_rate": 4.943701618578466e-05,
+      "loss": 0.3649,
+      "step": 4780
+    },
+    {
+      "epoch": 1.5523932729624839,
+      "grad_norm": 0.3246670961380005,
+      "learning_rate": 4.873328641801548e-05,
+      "loss": 0.3241,
+      "step": 4800
+    },
+    {
+      "epoch": 1.558861578266494,
+      "grad_norm": 0.41913464665412903,
+      "learning_rate": 4.802955665024631e-05,
+      "loss": 0.3486,
+      "step": 4820
+    },
+    {
+      "epoch": 1.5653298835705045,
+      "grad_norm": 0.38162580132484436,
+      "learning_rate": 4.7325826882477134e-05,
+      "loss": 0.3571,
+      "step": 4840
+    },
+    {
+      "epoch": 1.571798188874515,
+      "grad_norm": 0.34426695108413696,
+      "learning_rate": 4.6622097114707953e-05,
+      "loss": 0.3232,
+      "step": 4860
+    },
+    {
+      "epoch": 1.5782664941785254,
+      "grad_norm": 0.3782813847064972,
+      "learning_rate": 4.591836734693878e-05,
+      "loss": 0.346,
+      "step": 4880
+    },
+    {
+      "epoch": 1.5847347994825356,
+      "grad_norm": 0.42736709117889404,
+      "learning_rate": 4.52146375791696e-05,
+      "loss": 0.3275,
+      "step": 4900
+    },
+    {
+      "epoch": 1.5912031047865458,
+      "grad_norm": 0.37077978253364563,
+      "learning_rate": 4.4510907811400426e-05,
+      "loss": 0.3312,
+      "step": 4920
+    },
+    {
+      "epoch": 1.5976714100905562,
+      "grad_norm": 0.3196958303451538,
+      "learning_rate": 4.3807178043631246e-05,
+      "loss": 0.3156,
+      "step": 4940
+    },
+    {
+      "epoch": 1.6041397153945667,
+      "grad_norm": 0.36680254340171814,
+      "learning_rate": 4.3103448275862066e-05,
+      "loss": 0.3293,
+      "step": 4960
+    },
+    {
+      "epoch": 1.6106080206985771,
+      "grad_norm": 0.44455257058143616,
+      "learning_rate": 4.239971850809289e-05,
+      "loss": 0.3512,
+      "step": 4980
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "grad_norm": 0.37418290972709656,
+      "learning_rate": 4.169598874032372e-05,
+      "loss": 0.3422,
+      "step": 5000
+    },
+    {
+      "epoch": 1.6170763260025873,
+      "eval_loss": 0.5881822109222412,
+      "eval_runtime": 967.8912,
+      "eval_samples_per_second": 6.33,
+      "eval_steps_per_second": 0.791,
+      "step": 5000
+    },
+    {
+      "epoch": 1.6235446313065975,
+      "grad_norm": 0.3209955096244812,
+      "learning_rate": 4.0992258972554545e-05,
+      "loss": 0.3158,
+      "step": 5020
+    },
+    {
+      "epoch": 1.630012936610608,
+      "grad_norm": 0.31715771555900574,
+      "learning_rate": 4.0288529204785365e-05,
+      "loss": 0.3167,
+      "step": 5040
+    },
+    {
+      "epoch": 1.6364812419146184,
+      "grad_norm": 0.3968450725078583,
+      "learning_rate": 3.958479943701619e-05,
+      "loss": 0.3219,
+      "step": 5060
+    },
+    {
+      "epoch": 1.6429495472186288,
+      "grad_norm": 0.3187231123447418,
+      "learning_rate": 3.888106966924701e-05,
+      "loss": 0.3288,
+      "step": 5080
+    },
+    {
+      "epoch": 1.649417852522639,
+      "grad_norm": 0.35491153597831726,
+      "learning_rate": 3.817733990147783e-05,
+      "loss": 0.3298,
+      "step": 5100
+    },
+    {
+      "epoch": 1.6558861578266493,
+      "grad_norm": 0.4044147729873657,
+      "learning_rate": 3.747361013370866e-05,
+      "loss": 0.3505,
+      "step": 5120
+    },
+    {
+      "epoch": 1.6623544631306597,
+      "grad_norm": 0.33881333470344543,
+      "learning_rate": 3.676988036593948e-05,
+      "loss": 0.3269,
+      "step": 5140
+    },
+    {
+      "epoch": 1.6688227684346701,
+      "grad_norm": 0.3626604378223419,
+      "learning_rate": 3.6066150598170304e-05,
+      "loss": 0.3319,
+      "step": 5160
+    },
+    {
+      "epoch": 1.6752910737386806,
+      "grad_norm": 0.39335256814956665,
+      "learning_rate": 3.536242083040113e-05,
+      "loss": 0.3234,
+      "step": 5180
+    },
+    {
+      "epoch": 1.6817593790426908,
+      "grad_norm": 0.35490670800209045,
+      "learning_rate": 3.465869106263195e-05,
+      "loss": 0.3357,
+      "step": 5200
+    },
+    {
+      "epoch": 1.688227684346701,
+      "grad_norm": 0.303774356842041,
+      "learning_rate": 3.3954961294862776e-05,
+      "loss": 0.3609,
+      "step": 5220
+    },
+    {
+      "epoch": 1.6946959896507114,
+      "grad_norm": 0.3438192307949066,
+      "learning_rate": 3.3251231527093596e-05,
+      "loss": 0.3351,
+      "step": 5240
+    },
+    {
+      "epoch": 1.7011642949547219,
+      "grad_norm": 0.3261934518814087,
+      "learning_rate": 3.254750175932442e-05,
+      "loss": 0.3192,
+      "step": 5260
+    },
+    {
+      "epoch": 1.7076326002587323,
+      "grad_norm": 0.39429745078086853,
+      "learning_rate": 3.184377199155524e-05,
+      "loss": 0.3479,
+      "step": 5280
+    },
+    {
+      "epoch": 1.7141009055627425,
+      "grad_norm": 0.4267764985561371,
+      "learning_rate": 3.114004222378607e-05,
+      "loss": 0.3306,
+      "step": 5300
+    },
+    {
+      "epoch": 1.720569210866753,
+      "grad_norm": 0.3425467908382416,
+      "learning_rate": 3.0436312456016892e-05,
+      "loss": 0.3356,
+      "step": 5320
+    },
+    {
+      "epoch": 1.7270375161707632,
+      "grad_norm": 0.3064535856246948,
+      "learning_rate": 2.9732582688247712e-05,
+      "loss": 0.3341,
+      "step": 5340
+    },
+    {
+      "epoch": 1.7335058214747736,
+      "grad_norm": 0.3956790566444397,
+      "learning_rate": 2.902885292047854e-05,
+      "loss": 0.3375,
+      "step": 5360
+    },
+    {
+      "epoch": 1.739974126778784,
+      "grad_norm": 0.36128687858581543,
+      "learning_rate": 2.8325123152709358e-05,
+      "loss": 0.3395,
+      "step": 5380
+    },
+    {
+      "epoch": 1.7464424320827943,
+      "grad_norm": 0.33011215925216675,
+      "learning_rate": 2.7621393384940185e-05,
+      "loss": 0.3078,
+      "step": 5400
+    },
+    {
+      "epoch": 1.7529107373868047,
+      "grad_norm": 0.3741130530834198,
+      "learning_rate": 2.6917663617171008e-05,
+      "loss": 0.362,
+      "step": 5420
+    },
+    {
+      "epoch": 1.759379042690815,
+      "grad_norm": 0.319671094417572,
+      "learning_rate": 2.6213933849401827e-05,
+      "loss": 0.3105,
+      "step": 5440
+    },
+    {
+      "epoch": 1.7658473479948253,
+      "grad_norm": 0.4425293505191803,
+      "learning_rate": 2.5510204081632654e-05,
+      "loss": 0.3327,
+      "step": 5460
+    },
+    {
+      "epoch": 1.7723156532988358,
+      "grad_norm": 0.43660980463027954,
+      "learning_rate": 2.4806474313863477e-05,
+      "loss": 0.3057,
+      "step": 5480
+    },
+    {
+      "epoch": 1.7787839586028462,
+      "grad_norm": 0.3476182222366333,
+      "learning_rate": 2.41027445460943e-05,
+      "loss": 0.3362,
+      "step": 5500
+    },
+    {
+      "epoch": 1.7852522639068564,
+      "grad_norm": 0.37895113229751587,
+      "learning_rate": 2.3399014778325123e-05,
+      "loss": 0.337,
+      "step": 5520
+    },
+    {
+      "epoch": 1.7917205692108666,
+      "grad_norm": 0.38524091243743896,
+      "learning_rate": 2.2695285010555947e-05,
+      "loss": 0.362,
+      "step": 5540
+    },
+    {
+      "epoch": 1.798188874514877,
+      "grad_norm": 0.38090500235557556,
+      "learning_rate": 2.199155524278677e-05,
+      "loss": 0.3268,
+      "step": 5560
+    },
+    {
+      "epoch": 1.8046571798188875,
+      "grad_norm": 0.4094742238521576,
+      "learning_rate": 2.1287825475017596e-05,
+      "loss": 0.3353,
+      "step": 5580
+    },
+    {
+      "epoch": 1.811125485122898,
+      "grad_norm": 0.37963709235191345,
+      "learning_rate": 2.058409570724842e-05,
+      "loss": 0.3197,
+      "step": 5600
+    },
+    {
+      "epoch": 1.8175937904269082,
+      "grad_norm": 0.4362301230430603,
+      "learning_rate": 1.988036593947924e-05,
+      "loss": 0.3405,
+      "step": 5620
+    },
+    {
+      "epoch": 1.8240620957309184,
+      "grad_norm": 0.3662102520465851,
+      "learning_rate": 1.9176636171710062e-05,
+      "loss": 0.3262,
+      "step": 5640
+    },
+    {
+      "epoch": 1.8305304010349288,
+      "grad_norm": 0.3674371838569641,
+      "learning_rate": 1.847290640394089e-05,
+      "loss": 0.3394,
+      "step": 5660
+    },
+    {
+      "epoch": 1.8369987063389392,
+      "grad_norm": 0.31638914346694946,
+      "learning_rate": 1.7769176636171712e-05,
+      "loss": 0.3072,
+      "step": 5680
+    },
+    {
+      "epoch": 1.8434670116429497,
+      "grad_norm": 0.39859539270401,
+      "learning_rate": 1.7065446868402535e-05,
+      "loss": 0.3398,
+      "step": 5700
+    },
+    {
+      "epoch": 1.84993531694696,
+      "grad_norm": 0.37528204917907715,
+      "learning_rate": 1.6361717100633358e-05,
+      "loss": 0.3362,
+      "step": 5720
+    },
+    {
+      "epoch": 1.85640362225097,
+      "grad_norm": 0.40589115023612976,
+      "learning_rate": 1.565798733286418e-05,
+      "loss": 0.3369,
+      "step": 5740
+    },
+    {
+      "epoch": 1.8628719275549805,
+      "grad_norm": 0.3010273277759552,
+      "learning_rate": 1.4954257565095003e-05,
+      "loss": 0.3293,
+      "step": 5760
+    },
+    {
+      "epoch": 1.869340232858991,
+      "grad_norm": 0.47433990240097046,
+      "learning_rate": 1.4250527797325827e-05,
+      "loss": 0.3137,
+      "step": 5780
+    },
+    {
+      "epoch": 1.8758085381630014,
+      "grad_norm": 0.3495091199874878,
+      "learning_rate": 1.354679802955665e-05,
+      "loss": 0.3077,
+      "step": 5800
+    },
+    {
+      "epoch": 1.8822768434670116,
+      "grad_norm": 0.33797210454940796,
+      "learning_rate": 1.2843068261787475e-05,
+      "loss": 0.3454,
+      "step": 5820
+    },
+    {
+      "epoch": 1.8887451487710218,
+      "grad_norm": 0.34069564938545227,
+      "learning_rate": 1.2139338494018297e-05,
+      "loss": 0.3087,
+      "step": 5840
+    },
+    {
+      "epoch": 1.8952134540750323,
+      "grad_norm": 0.4089032709598541,
+      "learning_rate": 1.1435608726249122e-05,
+      "loss": 0.3487,
+      "step": 5860
+    },
+    {
+      "epoch": 1.9016817593790427,
+      "grad_norm": 0.3203008472919464,
+      "learning_rate": 1.0731878958479945e-05,
+      "loss": 0.3301,
+      "step": 5880
+    },
+    {
+      "epoch": 1.9081500646830531,
+      "grad_norm": 0.3505829870700836,
+      "learning_rate": 1.0028149190710768e-05,
+      "loss": 0.3319,
+      "step": 5900
+    },
+    {
+      "epoch": 1.9146183699870634,
+      "grad_norm": 0.29925537109375,
+      "learning_rate": 9.324419422941591e-06,
+      "loss": 0.3317,
+      "step": 5920
+    },
+    {
+      "epoch": 1.9210866752910736,
+      "grad_norm": 0.4060503840446472,
+      "learning_rate": 8.620689655172414e-06,
+      "loss": 0.3393,
+      "step": 5940
+    },
+    {
+      "epoch": 1.927554980595084,
+      "grad_norm": 0.4205261170864105,
+      "learning_rate": 7.916959887403237e-06,
+      "loss": 0.3375,
+      "step": 5960
+    },
+    {
+      "epoch": 1.9340232858990944,
+      "grad_norm": 0.44757845997810364,
+      "learning_rate": 7.213230119634061e-06,
+      "loss": 0.3329,
+      "step": 5980
+    },
+    {
+      "epoch": 1.9404915912031049,
+      "grad_norm": 0.5326984524726868,
+      "learning_rate": 6.509500351864884e-06,
+      "loss": 0.342,
+      "step": 6000
+    },
+    {
+      "epoch": 1.9404915912031049,
+      "eval_loss": 0.5839738845825195,
+      "eval_runtime": 967.7859,
+      "eval_samples_per_second": 6.331,
+      "eval_steps_per_second": 0.791,
+      "step": 6000
+    },
+    {
+      "epoch": 1.946959896507115,
+      "grad_norm": 0.36838072538375854,
+      "learning_rate": 5.8057705840957075e-06,
+      "loss": 0.3123,
+      "step": 6020
+    },
+    {
+      "epoch": 1.9534282018111255,
+      "grad_norm": 0.34141698479652405,
+      "learning_rate": 5.102040816326531e-06,
+      "loss": 0.3292,
+      "step": 6040
+    },
+    {
+      "epoch": 1.9598965071151357,
+      "grad_norm": 0.3648461103439331,
+      "learning_rate": 4.398311048557354e-06,
+      "loss": 0.3078,
+      "step": 6060
+    },
+    {
+      "epoch": 1.9663648124191462,
+      "grad_norm": 0.32175323367118835,
+      "learning_rate": 3.6945812807881777e-06,
+      "loss": 0.3033,
+      "step": 6080
+    },
+    {
+      "epoch": 1.9728331177231566,
+      "grad_norm": 0.31717827916145325,
+      "learning_rate": 2.990851513019001e-06,
+      "loss": 0.3028,
+      "step": 6100
+    },
+    {
+      "epoch": 1.9793014230271668,
+      "grad_norm": 0.5603770613670349,
+      "learning_rate": 2.287121745249824e-06,
+      "loss": 0.3414,
+      "step": 6120
+    },
+    {
+      "epoch": 1.9857697283311773,
+      "grad_norm": 0.36055296659469604,
+      "learning_rate": 1.5833919774806475e-06,
+      "loss": 0.3202,
+      "step": 6140
+    },
+    {
+      "epoch": 1.9922380336351875,
+      "grad_norm": 0.43246975541114807,
+      "learning_rate": 8.796622097114708e-07,
+      "loss": 0.3382,
+      "step": 6160
+    },
+    {
+      "epoch": 1.998706338939198,
+      "grad_norm": 0.3947717249393463,
+      "learning_rate": 1.7593244194229417e-07,
+      "loss": 0.318,
+      "step": 6180
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 6184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.235501163189567e+18,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

ultival_mistral/checkpoint-6184/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb8e5db938f70ff2ddfe787ade649d14bb55cc2be7a65bd63a20c3303bf68492
+size 5368

ultival_mistral/runs/Jul03_16-12-32_a183f886ff78/events.out.tfevents.1751559215.a183f886ff78.1482.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b120d0ce6d01806a33ad7290440a5809ea4d1b07201b096bee144699018b9b8a
+size 72589