Upload folder using huggingface_hub

README.md

@@ -1,51 +1,27 @@
 ---
-model-index:
-  - name: EEVE-Instruct-Math-10.8B
-    results:
-      - task:
-          type: text-generation
-        dataset:
-          name: gsm8k-ko
-          type: gsm8k
-        metrics:
-          - name: pass@1
-            type: pass@1
-            value: 0.4951
-            verified: false
 base_model:
 - yanolja/EEVE-Korean-Instruct-10.8B-v1.0
-- kuotient/EEVE-Math-10.8B-SFT
+- yanolja/EEVE-Korean-10.8B-v1.0
+- kuotient/EEVE-Math-10.8B
 library_name: transformers
 tags:
+- mergekit
 - merge
 
 ---
-# EEVE-Instruct-Math-10.8B
+# merge
 
-`EEVE-Math` 프로젝트는
-- Orca-Math-200k 번역 ([Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://arxiv.org/pdf/2402.14830.pdf))
-- gsm8k 번역, lm_eval 활용
-- Mergekit을 이용한 dare-ties 사용 ([DARE](https://arxiv.org/abs/2311.03099))
-
-에 대한 내용을 포괄하고 있습니다.
-
-> 이 모델은 EEVE-Math와 EEVE-Instruct의 dare-ties로 병합한 병합 모델입니다. 이 프로젝트는 이런 과정을 통해 특화 모델의 EEVE-Math의 성능을 많이 잃지 않고 Instruct 모델의 사용성을 유지할 수 있음을 보여주는 Proof of concept의 성격을 가지고 있습니다.
-
-| Model | gsm8k-ko(pass@1) |
-|---|---|
-| EEVE(Base) | 0.4049 |
-| [EEVE-Math](https://huggingface.co/kuotient/EEVE-Math-10.8B) (epoch 1) | 0.508 |
-| EEVE-Math (epoch 2) | **0.539** |
-| [EEVE-Instruct](https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0) | 0.4511 |
-| EEVE-Instruct + Math(epoch 1) | 0.4951 |
-| EEVE-Instruct + Math(epoch 2) | **0.5148** |
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
 
 ## Merge Details
-This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [yanolja/EEVE-Korean-Instruct-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0) as a base.
+### Merge Method
+
+This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [yanolja/EEVE-Korean-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-10.8B-v1.0) as a base.
 
 ### Models Merged
 
 The following models were included in the merge:
+* [yanolja/EEVE-Korean-Instruct-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0)
 * [kuotient/EEVE-Math-10.8B](https://huggingface.co/kuotient/EEVE-Math-10.8B)
 
 ### Configuration
@@ -54,37 +30,20 @@ The following YAML configuration was used to produce this model:
 
 ```yaml
 models:
-  - model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
+  - model: yanolja/EEVE-Korean-10.8B-v1.0
     # no parameters necessary for base model
-  - model: kuotient/EEVE-Math-10.8B
+  - model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
     parameters:
-      density: 1
+      density: 0.53
       weight: 0.6
+  - model: kuotient/EEVE-Math-10.8B
+    parameters:
+      density: 0.53
+      weight: 0.4
 merge_method: dare_ties
-base_model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
+base_model: yanolja/EEVE-Korean-10.8B-v1.0
 parameters:
   int8_mask: true
 dtype: bfloat16
-```
 
-## Evaluation
-[gsm8k-ko](https://huggingface.co/datasets/kuotient/gsm8k-ko), kobest
-```
-git clone https://github.com/kuotient/lm-evaluation-harness.git
-cd lm-evaluation-harness
-pip install -e .
 ```
-```
-lm_eval --model hf \
-    --model_args pretrained=yanolja/EEVE-Korean-Instruct-2.8B-v1.0 \
-    --tasks gsm8k-ko \
-    --device cuda:0 \
-    --batch_size auto:4
-```
-
-| Model | gsm8k(pass@1) | boolq(acc) | copa(acc) | hellaswag(acc) | Overall |
-|---|---|---|---|---|---|
-| yanolja/EEVE-Korean-10.8B-v1.0 | 0.4049 | - | - | - | - | - |
-| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 0.4511 | **0.8668** | **0.7450** | **0.4940** | 0.6392 |
-| [**EEVE-Math-10.8B**](https://huggingface.co/kuotient/EEVE-Math-10.8B) | **0.5390** | 0.8027 | 0.7260 | 0.4760 | 0.6359 |
-| **EEVE-Instruct-Math-10.8B** | 0.4951 | 0.8283 | 0.7500 | 0.4880 | **0.6403** |

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "yanolja/EEVE-Korean-Instruct-10.8B-v1.0",
+  "_name_or_path": "yanolja/EEVE-Korean-10.8B-v1.0",
   "architectures": [
     "LlamaForCausalLM"
   ],

mergekit_config.yml

@@ -1,12 +1,16 @@
 models:
-  - model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
+  - model: yanolja/EEVE-Korean-10.8B-v1.0
     # no parameters necessary for base model
-  - model: kuotient/EEVE-Math-10.8B-SFT
+  - model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
     parameters:
-      density: 1
+      density: 0.53
       weight: 0.6
+  - model: kuotient/EEVE-Math-10.8B
+    parameters:
+      density: 0.53
+      weight: 0.4
 merge_method: dare_ties
-base_model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
+base_model: yanolja/EEVE-Korean-10.8B-v1.0
 parameters:
   int8_mask: true
 dtype: bfloat16

model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0fc615af07a9208d6b93211c980fd97a05057455c30deb236988d46f87a7b551
+oid sha256:2f3c0f5a8b822a00ce02dba7b5501155e0ba361a89e43033dad5808e62e29166
 size 9898924776

model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b22e2f03056393d4f8f6c920c1980fd4dd82b51df20a24c4465d77d1eae22a44
+oid sha256:908c9149709e701103ea90b6eb46e306c64768c4780c0f6f859f508320718d03
 size 9831848976

model-00003-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ce55c3d6aa92c0a2216cc9cc87ae2d3fe1ce030025578d77e6a82903b902691e
+oid sha256:a90104a8aeaa3f67074a68805f41f5886a908fb14dcf23e0e6dc6c95ba635e2d
 size 1879125728