magiccodingman/Qwen3-4B-Instruct-2507-MXFP4-Hybrid-GGUF

Qwen3 4B Instruct 2507 MXFP4 Hybrid GGUF

Dense model utilizing MXFP4_MOE with hybrid weights on a dense model. Achieving interesting results that show smaller file size, more TPS, and near lossless precision.

Use one of the 3 found magic models!

Stats compared against the standard Q8_0 (precision loss still compared to F16)

• MXFP4_MOE-Q6_K

  +10.4% TPS vs Q8, 0.0178% precision loss, 4.5% smaller than Q8

  (TLDR: I'm a scientist or something like that..)

---

• MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K

  12% smaller than Q8 • 415.24 TPS • 0.0547% precision loss

  (TLDR: The perfect balance)

---

• MXFP4_MOE-output_q6_K-router_gate_emb_q6_K

  9.5% smaller than Q8 • 427.22 TPS • 0.1051% precision loss

  (TLDR: gotta go fast)

---

This repository contains a set of hybrid MXFP4 quantized GGUF models designed to explore a surprising discovery:

A carefully targeted combination of MXFP4 + high-precision embeddings/output weights can deliver near-Q8 accuracy with Q4–Q6 level throughput and smaller file sizes than Q8.

Unlike pure MXFP4, which heavily degrades dense models. This hybrid method selectively protects tensors that matter most for semantic stability, while allowing MXFP4 to accelerate everything else.

This is experimental. And should be treated as such. I am more than encouraging people to use these models and leave feedback! Though precision loss seemed near lossless, did the hybrid models act strange in certain situations? Worse or better on some topics compared to the original model? Did it do better/worse overall on everything? I'd love to hear back from others!

---

The Magic Models - Use One Of These 3 Models!

Each of these models achieved:

File size reduction compared to the Q8_0

Better precision loss scores than the pure Q6_K

Achieving noticeably better TPS than a Q4_K_M

I have personally deemed these in the category of "Q7.5" quantization.

The following are the special models to note from what was created. Each of the 3 models shown below are being compared to the Q8 model.

MXFP4_MOE-Q6_K

(+10.4% TPS vs Q8, 0.0178% loss, 4.5% smaller than Q8)

Honestly, this model feels like Q8 with a gym membership. It's safe for scientific work, most stable for long context, and hits a sweet ~10% boost in TPS.

The following was the conversion script:

llama-quantize \
  --tensor-type token_embd.weight=Q6_K \
  --tensor-type output.weight=Q6_K \
  "Path_To_F16_GGUF.gguf" \
  "Path_To_GGUF.gguf" \
  mxfp4_moe

MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K

(12% smaller than Q8 • 415.24 TPS • 0.0547% loss)

This is the most balanced choice for real world use. It's a rock star in long context windows, great TPS, and decent model size savings compared to the Q8 model.

The following was the conversion script:

llama-quantize \
  --tensor-type token_embd.weight=Q6_K \
  --tensor-type output.weight=MXFP4 \
  --tensor-type 'router.*'=Q6_K \
  --tensor-type 'gate.*'=Q6_K \
  "Path_To_F16_GGUF.gguf" \
  "Path_To_GGUF.gguf" \
  mxfp4_moe

MXFP4_MOE-output_q6_K-router_gate_emb_q6_K

(9.5% smaller than Q8 • 427.22 TPS • 0.1051% loss)

This model is pure gremlin energy. It's goes fast. That's just what it do.

The following was the conversion script:

llama-quantize \
  --tensor-type token_embd.weight=Q6_K \
  --tensor-type output.weight=Q6_K \
  --tensor-type 'router.*'=Q6_K \
  --tensor-type 'gate.*'=Q6_K \
  "Path_To_F16_GGUF.gguf" \
  "Path_To_GGUF.gguf" \
  mxfp4_moe

---

MXFP4_MOE Hybrid Naming Scheme & Synopsis

Multiple different combinations of converted models were created. The results were interesting to say the least. The following table will explain my naming scheme to what was done to the model to create it.

Suffix Example	Meaning
MXFP4_MOE	Pure MXFP4 pipeline
MXFP4_MOE-Q8	Embedding/output in Q8_0
MXFP4_MOE-F16	Embedding/output in F16
output_mxfp4-embd_q8	Output → MXFP4, Embedding → Q8
output_mxfp4-router_gate_emb_q5_K	Output → MXFP4, Emb/Router/Gate → Q5_K
MXFP4_MOE-Q6_K	Both embedding + output in Q6_K
Q8_0, Q6_K, Q4_K_M	Pure model-wide quantizations

The results achieved were interesting to say the least. It was a brute force game of mass creating models with hybrid methods to find combinations that didn't cause too much noise and paired well with MXFP4.

This repo showcases the converted models, whether good or bad that was created. But, I have been testing other models in different combinations as well. The winning hybrid combinations shown in this repo DOES NOT always equate to the same results on different models.

Some models do better or worse with different kinds of combinations. It depends if it's dense, MOE, and much more. Many times the results surprise me. Many models no matter the combination will not play nice with MXFP4. At least with the methods shown here.

---

Benchmark Methodology

All models were tested with a unified automated harness using llama.cpp tools.

Included tests:

• Throughput:
  llama-bench with descending GPU offload (-ngl 35 → 0) and automatic OOM retry.
  Highest successful TPS is recorded.

• Perplexity:
  Three domains: general, code, math.
  Each uses an auto-generated corpus of ~32k tokens.
  Perplexity is computed with llama-perplexity at 2048-token context.
  Same GPU retry logic as above.

• Precision loss:
  Each model is compared to its family F16 baseline.
  Precision-loss % is computed for all PPL domains, plus an averaged score.
  Models are ranked by this metric.

---

Table - Overview of Results

Comparing to F16.

model_name	size_reduction	tps_change
Q8_0	46.8%	42.02%
MXFP4_MOE-Q8	46.8%	39.75%
MXFP4_MOE-Q6_K	49.2%	56.73%
MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K	53.2%	57.76%
MXFP4_MOE-F16	37.87%	7.49%
MXFP4_MOE-output_q6_K-router_gate_emb_q6_K	51.87%	62.31%
MXFP4_MOE-output_mxfp4-embd_q6_K	50.53%	59.39%
MXFP4_MOE-output_mxfp4-embd_q8	49.33%	44.6%
MXFP4_MOE-output_mxfp4-router_gate_emb_q8	49.33%	48.47%
MXFP4_MOE-output_mxfp4-embd_q5_K	51.2%	70.98%
Q6_K	58.93%	43.73%
MXFP4_MOE-Q5_K	50.4%	67.03%
MXFP4_MOE-output_mxfp4-router_gate_emb_q5_K	55.33%	67.74%
Q5_K_M	64.13%	43.3%
MXFP4_MOE-output_mxfp4-embd_q4_K	51.73%	76.42%
Q4_K_M	68.93%	54.38%
MXFP4_MOE-output_mxfp4-router_gate_emb_q4_K	57.33%	83.78%
MXFP4_MOE-Q4_K	51.6%	72.93%
MXFP4_MOE-output_q8-embd_mxfp4	49.47%	77.71%
MXFP4_MOE	73.33%	91.76%

• All percentages compared against the selected family F16 baseline.

---

Table - File Size + TPS + Avg Precision Loss

model_name	file_size_gb	bench_tps	avg_prec_loss
F16	7.5	263.21	0
Q8_0	3.99	373.82	0.0049
MXFP4_MOE-Q8	3.99	367.84	0.0126
MXFP4_MOE-Q6_K	3.81	412.52	0.0178
MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K	3.51	415.24	0.0547
MXFP4_MOE-F16	4.66	282.93	0.0647
MXFP4_MOE-output_q6_K-router_gate_emb_q6_K	3.61	427.22	0.1051
MXFP4_MOE-output_mxfp4-embd_q6_K	3.71	419.53	0.1371
MXFP4_MOE-output_mxfp4-embd_q8	3.8	380.59	0.1725
MXFP4_MOE-output_mxfp4-router_gate_emb_q8	3.8	390.79	0.1725
MXFP4_MOE-output_mxfp4-embd_q5_K	3.66	450.03	0.1873
Q6_K	3.08	378.3	0.2635
MXFP4_MOE-Q5_K	3.72	439.64	0.4048
MXFP4_MOE-output_mxfp4-router_gate_emb_q5_K	3.35	441.5	0.5808
Q5_K_M	2.69	377.17	0.7871
MXFP4_MOE-output_mxfp4-embd_q4_K	3.62	464.35	0.8979
Q4_K_M	2.33	406.34	0.9867
MXFP4_MOE-output_mxfp4-router_gate_emb_q4_K	3.2	483.72	1.2362
MXFP4_MOE-Q4_K	3.63	455.17	1.247
MXFP4_MOE-output_q8-embd_mxfp4	3.79	467.74	2.9366
MXFP4_MOE	2	504.74	8.2072

• Bench NGL was 35
• Utilized CUDA

---

Table - PPL Columns

model_name	gen	gen_er	code	code_er	math	math_er
F16	8.8841	0.2056	1.5468	0.0122	6.7111	0.137
Q8_0	8.8754	0.2053	1.5488	0.0123	6.708	0.1367
MXFP4_MOE-Q8	8.8803	0.2055	1.5486	0.0123	6.7087	0.1367
MXFP4_MOE-Q6_K	8.8802	0.2052	1.5484	0.0123	6.7107	0.1365
MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K	8.8822	0.2047	1.546	0.0121	6.705	0.1359
MXFP4_MOE-F16	8.8889	0.2058	1.548	0.0123	6.7153	0.137
MXFP4_MOE-output_q6_K-router_gate_emb_q6_K	8.8798	0.2053	1.548	0.0123	6.7303	0.1372
MXFP4_MOE-output_mxfp4-embd_q6_K	8.8785	0.2046	1.5463	0.0121	6.6899	0.1353
MXFP4_MOE-output_mxfp4-embd_q8	8.8753	0.2047	1.5457	0.0121	6.6878	0.1355
MXFP4_MOE-output_mxfp4-router_gate_emb_q8	8.8753	0.2047	1.5457	0.0121	6.6878	0.1355
MXFP4_MOE-output_mxfp4-embd_q5_K	8.9251	0.206	1.5466	0.0121	6.7187	0.1365
Q6_K	8.8441	0.2034	1.5452	0.0121	6.6952	0.1357
MXFP4_MOE-Q5_K	8.9263	0.2066	1.551	0.0123	6.7425	0.1378
MXFP4_MOE-output_mxfp4-router_gate_emb_q5_K	8.979	0.2081	1.5481	0.0121	6.7507	0.1375
Q5_K_M	8.9731	0.2079	1.554	0.0123	6.7711	0.1384
MXFP4_MOE-output_mxfp4-embd_q4_K	9.0428	0.2099	1.5503	0.0121	6.7568	0.1364
Q4_K_M	8.9569	0.2055	1.5694	0.0125	6.7567	0.1372
MXFP4_MOE-output_mxfp4-router_gate_emb_q4_K	9.0761	0.2108	1.5546	0.0121	6.7811	0.1372
MXFP4_MOE-Q4_K	9.0787	0.2116	1.5573	0.0124	6.7696	0.1372
MXFP4_MOE-output_q8-embd_mxfp4	9.2094	0.2117	1.5598	0.0123	7.0002	0.1427
MXFP4_MOE	9.8799	0.2282	1.6122	0.013	7.3275	0.1494

• gen = ppl_general
• gen_er = ppl_general_error
• code = ppl_code
• code_er = ppl_code_error
• math = ppl_math
• math_er = ppl_math_error

---

Table - Precision Loss Columns

model_name	loss_general	loss_code	loss_math
F16	0	0	0
Q8_0	-0.0979	0.1293	-0.0462
MXFP4_MOE-Q8	-0.0428	0.1164	-0.0358
MXFP4_MOE-Q6_K	-0.0439	0.1034	-0.006
MXFP4_MOE-output_mxfp4-router_gate_emb_q6_K	-0.0214	-0.0517	-0.0909
MXFP4_MOE-F16	0.054	0.0776	0.0626
MXFP4_MOE-output_q6_K-router_gate_emb_q6_K	-0.0484	0.0776	0.2861
MXFP4_MOE-output_mxfp4-embd_q6_K	-0.063	-0.0323	-0.3159
MXFP4_MOE-output_mxfp4-embd_q8	-0.0991	-0.0711	-0.3472
MXFP4_MOE-output_mxfp4-router_gate_emb_q8	-0.0991	-0.0711	-0.3472
MXFP4_MOE-output_mxfp4-embd_q5_K	0.4615	-0.0129	0.1132
Q6_K	-0.4502	-0.1034	-0.2369
MXFP4_MOE-Q5_K	0.475	0.2715	0.4679
MXFP4_MOE-output_mxfp4-router_gate_emb_q5_K	1.0682	0.084	0.5901
Q5_K_M	1.0018	0.4655	0.894
MXFP4_MOE-output_mxfp4-embd_q4_K	1.7863	0.2263	0.681
Q4_K_M	0.8194	1.4611	0.6795
MXFP4_MOE-output_mxfp4-router_gate_emb_q4_K	2.1612	0.5043	1.043
MXFP4_MOE-Q4_K	2.1904	0.6788	0.8717
MXFP4_MOE-output_q8-embd_mxfp4	3.6616	0.8404	4.3078
MXFP4_MOE	11.2088	4.2281	9.1848

• loss_general = precision_loss_general_pct
• loss_code = precision_loss_code_pct
• loss_math = precision_loss_math_pct

---

Takeaway

• I found it very interesting how in general the Q6_K hybrid played nicer with MXFP4 than the Q8.
• I didn't include the MXFP4 Q8 in my suggested "good batch" category. Because it was the same file size, achieving the same or worse TPS, while losing precision compared to the normal Q8. So there was no real advantages to that model, even if it beat the chosen MXFP4 Q6_K hybrid models that I deemed as "Q7.5" category.
• Leaving the output weights on the conversion on MXFP4 consistently improved TPS with minimal precision loss. But in any scenario where the embd was left as MXFP4, it'd result in significantly lower precision.
• The pure MXFP4_MOE had significant brain damage. The AI model was NOT okay.
• The models I provided in the Bad-Hybrid-Models are the models I've deemed, "not useful enough to utilize compared to their normal non quantized counter parts."