Is this MLC LLM quantized or GPTQ?
#1
by
MLDataScientist
- opened
HI,
It seems this quantization is in MLC LLM format. Can you please change the name and remove GPTQ from it? I was initially trying to download it but then I understood it was not GPTQ.
Can you please change the name and remove GPTQ from it?
ICYMI, GPTQ is a quantization algorithm. This model is a model that has been quantized with AutoGPTQ, serialized in the mlc-llm format.
numen-tech
changed discussion status to
closed
Oh I see. Thanks for clarification. So, this model will run only with mlc-llm backend, not vllm GPTQ backend, right?
Oh I see. Thanks for clarification. So, this model will run only with mlc-llm backend, not vllm GPTQ backend, right?
Indeed!