Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers excluding "lm_head", "*.gate"
  • Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • KV Cache: FP8 symmetric per-tensor

• Quick Start

1. Download and install Quark
2. [TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

The Quark quantized Auto Mixed Precision (AMP) models are now supported to be easily deployed in vLLM backend (vLLM-compatible).

Evaluation

The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

Quant scheme	arc challenge (↑) (acc)		gsm8k (↑) (strict-match)		mmlu (↑) (acc)		wikitext (↓) (word_perplexity)		winogrande (↑) (acc)
	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate
FP16	0.6271	100.0%	0.6384	100.0%	0.6893	100.0%	5.7259	100.0%	0.7664	100.0%
FP8	0.6220	99.2%	0.6384	100.0%	0.6816	98.9%	5.8092	98.6%	0.7830	102.2%
AMP	0.6237	99.5%	0.6133	96.1%	0.6765	98.1%	6.2020	92.3%	0.7569	98.8%
MXFP4	0.5922	94.4%	0.4845	75.9%	0.6364	92.3%	7.0935	80.7%	0.7474	97.5%

License

Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.