Model save

README.md

@@ -0,0 +1,78 @@
+---
+license: gemma
+library_name: peft
+tags:
+- trl
+- sft
+- generated_from_trainer
+base_model: google/gemma-7b
+datasets:
+- generator
+model-index:
+- name: gemma7b-coding-gpt4o-100k
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# gemma7b-coding-gpt4o-100k
+
+This model is a fine-tuned version of [google/gemma-7b](https://huggingface.co/google/gemma-7b) on the generator dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.9658
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 32
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 10
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 0.5262        | 0.9989 | 470  | 1.3224          |
+| 0.4826        | 2.0    | 941  | 1.3435          |
+| 0.4369        | 2.9989 | 1411 | 1.4787          |
+| 0.3819        | 4.0    | 1882 | 1.7432          |
+| 0.3345        | 4.9989 | 2352 | 2.1234          |
+| 0.2875        | 6.0    | 2823 | 2.5846          |
+| 0.2319        | 6.9989 | 3293 | 3.1057          |
+| 0.1968        | 8.0    | 3764 | 3.6609          |
+| 0.1809        | 8.9989 | 4234 | 3.9400          |
+| 0.1757        | 9.9894 | 4700 | 3.9658          |
+
+
+### Framework versions
+
+- PEFT 0.11.1
+- Transformers 4.40.1
+- Pytorch 2.2.0+cu121
+- Datasets 2.19.2
+- Tokenizers 0.19.1

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a01770e492d5007c7493094cb6180bcf753cb3ea310c17f8de78ebe1b65b2a64
+oid sha256:8cd85a50c9d7598b9f01e04ebe8808bfb48a595e54a385ebec95f9db180b43e5
 size 200068904

all_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 9.989373007438894,
+    "total_flos": 7.255013733871649e+18,
+    "train_loss": 0.5076825852977469,
+    "train_runtime": 24662.4426,
+    "train_samples": 92048,
+    "train_samples_per_second": 6.102,
+    "train_steps_per_second": 0.191
+}

runs/Jun12_06-38-11_b9de806b912b/events.out.tfevents.1718175163.b9de806b912b.2859.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:53a9912f31dc95e17f9a9f1aef450b335b5d730fe9c6338a86241a83d144fdf2
-size 202075
+oid sha256:8e6b9a8d53217cd81ce16c3dbcbcd5f709083d119a415603902766d69adb6a55
+size 206920

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 9.989373007438894,
+    "total_flos": 7.255013733871649e+18,
+    "train_loss": 0.5076825852977469,
+    "train_runtime": 24662.4426,
+    "train_samples": 92048,
+    "train_samples_per_second": 6.102,
+    "train_steps_per_second": 0.191
+}