|
|
--- |
|
|
base_model: |
|
|
- meta-llama/Llama-2-13b-hf |
|
|
base_model_relation: quantized |
|
|
license: llama2 |
|
|
--- |
|
|
# Model Card |
|
|
|
|
|
- Base model: `meta-llama/Llama-2-13b-hf` |
|
|
- Quantization method: LNQ with GuidedQuant Hessian |
|
|
- Target bit-width: 4 |
|
|
- Backend kernel: Any-Precision-LLM kernel (`ap-gemv`) |
|
|
- Calibration data: RedPajama (1024 sentences / 4096 tokens) |
|
|
- Calibration objective: Next-token prediction |
|
|
- num_groups (for GuidedQuant Hessian): 4 |
|
|
|
|
|
# How to run |
|
|
- Follow the instruction in https://github.com/snu-mllab/GuidedQuant. |
|
|
|
|
|
# References |
|
|
- [Model Paper](https://arxiv.org/abs/2505.07004) |
