Update README.md
Browse files
README.md
CHANGED
|
@@ -15,9 +15,9 @@ tags:
|
|
| 15 |
- auto-gptq
|
| 16 |
- intel
|
| 17 |
license: apache-2.0
|
| 18 |
-
model_name: Minerva
|
| 19 |
base_model:
|
| 20 |
-
- sapienzanlp/Minerva-
|
| 21 |
inference: false
|
| 22 |
model_creator: sapienzanlp
|
| 23 |
datasets:
|
|
@@ -30,7 +30,7 @@ quantized_by: fbaldassarri
|
|
| 30 |
|
| 31 |
## Model Information
|
| 32 |
|
| 33 |
-
Quantized version of [sapienzanlp/Minerva-
|
| 34 |
- 8 bits (INT8)
|
| 35 |
- group size = 128
|
| 36 |
- Symmetrical Quantization
|
|
@@ -38,7 +38,7 @@ Quantized version of [sapienzanlp/Minerva-1B-base-v1.0](https://huggingface.co/s
|
|
| 38 |
|
| 39 |
Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round) v0.4.6
|
| 40 |
|
| 41 |
-
Note: this INT8 version of Minerva-
|
| 42 |
|
| 43 |
## Replication Recipe
|
| 44 |
|
|
@@ -63,14 +63,14 @@ pip install -vvv --no-build-isolation -e .[cpu]
|
|
| 63 |
|
| 64 |
```
|
| 65 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 66 |
-
model_name = "sapienzanlp/Minerva-
|
| 67 |
model = AutoModelForCausalLM.from_pretrained(model_name)
|
| 68 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 69 |
from auto_round import AutoRound
|
| 70 |
bits, group_size, sym, device, amp = 8, 128, True, 'cpu', False
|
| 71 |
autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym, device=device, amp=amp)
|
| 72 |
autoround.quantize()
|
| 73 |
-
output_dir = "./AutoRound/sapienzanlp_Minerva-
|
| 74 |
autoround.save_quantized(output_dir, format='auto_gptq', inplace=True)
|
| 75 |
```
|
| 76 |
|
|
|
|
| 15 |
- auto-gptq
|
| 16 |
- intel
|
| 17 |
license: apache-2.0
|
| 18 |
+
model_name: Minerva 350M base v1.0
|
| 19 |
base_model:
|
| 20 |
+
- sapienzanlp/Minerva-350M-base-v1.0
|
| 21 |
inference: false
|
| 22 |
model_creator: sapienzanlp
|
| 23 |
datasets:
|
|
|
|
| 30 |
|
| 31 |
## Model Information
|
| 32 |
|
| 33 |
+
Quantized version of [sapienzanlp/Minerva-350M-base-v1.0](https://huggingface.co/sapienzanlp/Minerva-350M-base-v1.0) using torch.float32 for quantization tuning.
|
| 34 |
- 8 bits (INT8)
|
| 35 |
- group size = 128
|
| 36 |
- Symmetrical Quantization
|
|
|
|
| 38 |
|
| 39 |
Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round) v0.4.6
|
| 40 |
|
| 41 |
+
Note: this INT8 version of Minerva-350M-base-v1.0 has been quantized to run inference through CPU.
|
| 42 |
|
| 43 |
## Replication Recipe
|
| 44 |
|
|
|
|
| 63 |
|
| 64 |
```
|
| 65 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 66 |
+
model_name = "sapienzanlp/Minerva-350M-base-v1.0"
|
| 67 |
model = AutoModelForCausalLM.from_pretrained(model_name)
|
| 68 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 69 |
from auto_round import AutoRound
|
| 70 |
bits, group_size, sym, device, amp = 8, 128, True, 'cpu', False
|
| 71 |
autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym, device=device, amp=amp)
|
| 72 |
autoround.quantize()
|
| 73 |
+
output_dir = "./AutoRound/sapienzanlp_Minerva-350M-base-v1.0-autogptq-int8-gs128-sym"
|
| 74 |
autoround.save_quantized(output_dir, format='auto_gptq', inplace=True)
|
| 75 |
```
|
| 76 |
|