SeerAttention/SeerAttention-Qwen2.5-32B-AttnGates

This repo only contains the AttnGates' weights for Qwen2.5-32B-Instruct Model.

SeerAttention introduces learnable AttnGate modules to accelerate the computationally intensive prefill stage of long-context large language models (LLMs) via dynamic block-level sparsity. The AttnGates are trained in a parameter-efficient self-distillation framework, where they learn to mimic the 2D max-pooled attention patterns of the original frozen model, preserving its integrity while avoiding costly retraining. During inference, these gates generate block-sparse binary masks by applying threshold/TopK to their learned soft scores, enabling efficient computation through a custom block-sparse FlashAttention kernel.

Original Github Repo

https://github.com/microsoft/SeerAttention.

Evaluation Results

PG19 PPL

density	8192	16384	32768
0.1	8.11	7.76	7.72
0.2	7.85	7.62	7.62
0.3	7.77	7.58	7.59
0.4	7.75	7.57	7.58
0.5	7.73	7.56	7.57
1.0	7.72	7.55	7.57

LongBench

Task	0-4k (Dense / Sparse)	4-8k (Dense / Sparse)	8k+ (Dense / Sparse)
hotpotqa	74.73 / 75.73	66.92 / 67.28	66.05 / 65.59
trec	68.00 / 68.00	79.00 / 78.00	80.00 / 80.00
2wikimqa	71.01 / 71.01	61.59 / 61.26	49.36 / 49.59
multi_news	23.60 / 23.37	21.09 / 21.12	20.55 / 20.55
lcc	58.20 / 58.84	52.76 / 50.60	53.98 / 54.57
qasper	50.23 / 50.25	38.80 / 38.72	38.48 / 39.22
passage_count	31.00 / 31.00	18.00 / 18.00	16.00 / 20.00
passage_retrieval_en	100.0 / 100.0	100.0 / 99.00	99.00 / 99.00
triviaqa	84.68 / 84.68	88.79 / 89.42	86.37 / 85.43
samsum	41.16 / 41.26	41.13 / 41.65	46.88 / 46.36
gov_report	29.90 / 30.09	30.70 / 30.91	29.35 / 29.46
repobench-p	42.98 / 42.90	32.73 / 33.25	36.82 / 35.37
multifieldqa_en	56.26 / 56.51	46.73 / 45.86	50.99 / 50.99
averaged score	56.29 / 56.43	52.17 / 51.93	51.83 / 52.01
averaged density	0.895	0.682	0.409

LongBenchV2 CoT Benchmark

All the SeerAttention models run with threshold=5e-4.

For R1-Distilled models, we remove the two passes generation setup (think + summary), we directly ask the models to output anwser after thinking. The generation max length is set to 10240.

Model	Overall	Easy	Hard	Short	Medium	Long
Llama-3.1-8B-Instruct	30.4	31.2	29.9	37.8	24.7	29.6
SeerAttention-Llama-3.1-8B	31.6	33.3	30.5	33.9	31.6	27.8
Qwen2.5-14B-Instruct	34.8	37.5	33.1	44.4	32.1	24.1
SeerAttention-Qwen2.5-14B	32.8	38.0	29.6	45.0	30.2	17.6
Qwen2.5-32B-Instruct	36.4	42.2	32.8	47.8	29.8	30.6
SeerAttention-Qwen2.5-32B	36.4	41.1	33.4	49.4	29.8	27.8
DeepSeek-R1-Distill-Qwen-14B	34.2	43.2	28.6	45.0	27.9	28.7
SeerAttention-DeepSeek-R1-Distill-Qwen-14B	31.6	35.9	28.9	41.7	26.0	25.9
DeepSeek-R1-Distill-Qwen-32B	37.2	42.7	33.8	47.2	35.8	23.1
SeerAttention-DeepSeek-R1-Distill-Qwen-32B	37.0	42.2	33.8	49.4	31.6	26.9