tokenlabsdotrun/Llama-3.1-8B-Quanto-Int8

Llama-3.1-8B-Instruct Quantized (Quanto INT8)

This is a quantized version of meta-llama/Llama-3.1-8B-Instruct using quanto with INT8 weight quantization.

Model Details

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Quantization Method: quanto
• Weight Precision: INT8 (qint8)
• Activation Precision: Original (bfloat16)
• Original Size: ~16 GB (bfloat16)
• Quantized Size: ~8.5 GB

Quantization Benefits

• ~50% memory reduction compared to bfloat16
• Faster inference on CPU
• Minimal quality degradation for most tasks

Usage

Loading the quantized model

from transformers import AutoModelForCausalLM, AutoTokenizer
from quanto import safe_load, freeze, quantize, qint8

# Load base model structure
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Quantize and load weights
quantize(model, weights=qint8)
state_dict = safe_load("model.safetensors")
model.load_state_dict(state_dict)
freeze(model)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("tokenlabsdotrun/Llama-3.1-8B-Quanto-Int8")

# Generate text
inputs = tokenizer("Hello, my name is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• Quanto quantized models require the quanto library to load
• Slight accuracy loss compared to full precision model
• Best suited for inference, not fine-tuning

License

This model inherits the Llama 3.1 Community License.

Acknowledgments

• Meta AI for the original Llama 3.1 model
• Hugging Face for the quanto quantization library