iproskurina/Mistral-7B-v0.3-awq-int4

Mistral-7B-v0.3 - AWQ (4-bit)

Source model: mistralai/Mistral-7B-v0.3

This model was quantized to 4-bit using VLLM-Compressor.

Quantization parameters: 4-bit, symmetric scheme.

Usage

# pip install vllm
from vllm import LLM
model = LLM("iproskurina/Mistral-7B-v0.3-awq-int4")
output = model.generate("The capital of France is")
print(output)```