Optimizing Qwen2.5-Coder Throughput with NVIDIA TensorRT-LLM Lookahead Decoding
Large language models (LLMs)
that specialize in coding have been steadily adopted into developer workflows. From pair programming to self-improving
AI agents
, these models assist developers with various tasks, including enhancing code, fixing bugs, generating tests, and writing documentation.
To promote the development of open-source LLMs, the Qwen team recently released Qwen2.5-Coder, a family of advanced LLMs for code generation, reasoning, and fixing across popular programming languages. This post explores the benefits of inference optimizations for Qwen2.5-Coder models supported in
NVIDIA TensorRT-LLM
, and the ease of deployment with
NVIDIA NIM
for transformative potential and coding efficiency.
Qwen2.5-Coder models
The Qwen2.5-Coder models have achieved state-of-the-art performance across popular academic benchmarks.
NVIDIA TensorRT-LLM
has optimized three popular models from the Qwen2.5-Coder family—the 1.5B, 7B, and 32B versions—for high throughput and low latency. TensorRT-LLM is a library for fast, efficient LLM inference and includes optimizations such as dynamic
inflight batching
,
KV caching
,
KV cache reuse
, and several speculative decoding techniques, among others.
These optimizations help deliver performance improvements for the Qwen2.5-Coder models on popular programming languages such as Python, C++, Java, Bash, Javascript, TypeScript, and Go, reaching a wider range of developers. This post explores the lookahead decoding optimization and the performance boost it helps achieve. Without any additional training or need for additional draft models, developers can leverage the TensorRT-LLM
high-level API
to speed up Qwen2.5-Coder inference to generate multiline autocode completion.
Lookahead decoding
Lookahead decoding
is a speculative decoding technique that addresses the slow autoregressive nature of LLMs. Each autoregressive decoding step only generates one token at a time, not leveraging the massive parallel processing power of NVIDIA GPUs, leading to low GPU utilization and lower throughput. We’ve previously discussed the throughput boost achievable with
draft target speculative decoding
, and here we discuss the benefits of leveraging TensorRT-LLM lookahead decoding implementation using the Qwen2.5-Coder models as an example.
Unlike the single-token generation in autoregressive decoding, lookahead decoding generates multiple tokens simultaneously, adequately utilizing the parallel processing capabilities of the GPU, leveraging computation (FLOPs) for latency reduction. Moreover, lookahead decoding doesn’t require a separate draft model that’s needed for draft target speculative decoding.
Each decoding step is divided into two parallel branches, the lookahead branch and the verification branch. Using the
Jacobi iteration method
, a classic nonlinear systems solver, the lookhead branch performs parallel decoding for future tokens by generating n-grams. The verification branch selects and verifies the promising n-gram candidates generated by the lookahead branch.
The lookahead algorithm is configured using three key parameters: window size (W), n-gram size (N), and verification set size (G).
Window size (W):
Represents the lookahead window size, which determines how many future tokens the algorithm attempts to predict in each step. Larger window size enables the model to look further, helping generate more tokens in a single pass. This effectively improves throughput performance while utilizing GPU computation FLOPs efficiently.
N-gram size (N):
Represents the size of the n-grams used in the lookahead process. For example, a 5-gram is a contiguous sequence of 5 future tokens. Together with the window size, it creates a fixed-sized, 2D window for the lookahead branch to generate n-grams from the Jacobi iteration trajectory.
Verification set size (G):
Represents the maximum number of speculations or candidate n-grams that the algorithm considers in each step for verification. It balances the trade-off between computation efficiency and exploring more possibilities.
Figure 1. Lookahead decoding workflow with (W, N, G) = (5, 3, 2). Image credit:
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Lookahead performance greatly depends on the base model, hardware, batch size, sequence length, and the dataset. It is recommended to profile various configurations to find the best (W, N, G) configuration given the setup. Optimal (W, N, G) tuple configuration enables lookahead decoding to deliver improved throughput performance without the need for any additional training, fine-tuning or draft models.
Through our experiments on (W, N, G) configuration values sweep, we achieve 3.6x and 1.6x throughput speedups for Qwen2.5-Coder 7B Instruct and Qwen2.5-Coder 32B Instruct models, respectively. These speedups are measured in throughput (tokens/second) compared to baseline (no lookahead speculative decoding) on
NVIDIA H100 Tensor Core GPUs
, as shown in Figure 2.
Figure 2. Qwen2.5-Coder models throughput boost on NVIDIA DGX H100 with TensorRT-LLM lookahead decoding
Data measured on 01/30/2025. Inference throughput (output tokens/second) speedups of Qwen2.5-Coder 7B Instruct and Qwen2.5-Coder 32B Instruct models. DGX H100, TP=1 | (W, N, G) = (8, 8, 8) | Qwen2.5-Coder 7B Instruct, TP=2 | (W, N, G) = (15, 15, 15) | Qwen2.5-Coder-32B-Instruct, batch size=1, TensorRT-LLM version 0.15.0​.
Similar throughput speedups are achieved on
NVIDIA H200 Tensor Core GPUs
. With their higher memory bandwidth, they also help raise the baseline throughput performance leading to slightly lower speedups as compared to H100 GPUs (Figure 3).
Figure 3. Qwen2.5-Coder models throughput boost on NVIDIA DGX H200 with TensorRT-LLM lookahead decoding
Data measured on 01/30/2025. Inference throughput (output tokens/second) speedups of Qwen2.5-Coder 7B Instruct and Qwen2.5-Coder 32B Instruct models. DGX H200, TP=1 | (W, N, G) = (8, 8, 8) | Qwen2.5-Coder 7B Instruct, TP=2 | (W, N, G) = (15, 15, 15) | Qwen2.5-Coder 32B Instruct, batch size=1, TensorRT-LLM version 0.15.0​.
Steps to run lookahead decoding with TensorRT-LLM
To reproduce these performance gains using lookahead speculative decoding within TensorRT-LLM, follow the steps below.
# Install TensorRT-LLM. (Commands below are for Linux. Refer to TensorRT-LLM docs for Windows)

sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade setuptools 
&& pip3 install tensorrt_llm --extra-index-url https://pypi.nvidia.com
Then run lookahead decoding in TensorRT-LLM using the high-level API.
# Command for Qwen2.5-Coder-7B-Instruct

from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi import (LLM, BuildConfig, KvCacheConfig, 
LookaheadDecodingConfig, SamplingParams)
def main():
    """The end user can customize the build configuration with the 
build_config class. # Max draft length is based on (W,N,G) values and 
calculated as: (W + G -1) * (N-1) + ( N<=1 ? 0: N-2)"""

    build_config = BuildConfig(max_batch_size = 128,
max_input_len = 2048, 
max_seq_len = 4096,
max_num_tokens = 16384, 
max_draft_len = 111)
    build_config.plugin_config.reduce_fusion = True
    build_config.plugin_config.use_paged_context_fmha = True 
    build_config.plugin_config.multiple_profiles = True
    
    # The configuration for lookahead decoding
    lookahead_config = LookaheadDecodingConfig(max_window_size=8,
                                               max_ngram_size=8,
                                              max_verification_set_size=8)
    
    kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.4)
    llm = LLM(model="Qwen/Qwen2.5-Coder-7B-Instruct",
              kv_cache_config=kv_cache_config,
              build_config=build_config,
              speculative_config=lookahead_config)
    
    prompt = """Write a C++ program to find the nth Fibonacci number using 
recursion. Now we define a sequence of numbers in which each number is the 
sum of the three preceding ones. The first three numbers are 0, -1, -1. 
Write a program to find the nth number."""
    
    sampling_params = SamplingParams(lookahead_config=lookahead_config)
    
    output = llm.generate(prompt, sampling_params=sampling_params)
    print(output)
if __name__ == '__main__':
    main()
Summary
Lookahead speculative decoding enables throughput boost on LLMs without any additional training, fine-tuning, or draft models. We presented benchmarked performance improvements on Qwen2.5-Coder models. Visit build.nvidia.com to
try the Qwen2.5-Coder models
optimized with NVIDIA TensorRT-LLM for free. Qwen2.5-Coder models optimized with TensorRT-LLM have also been packaged as downloadable
NVIDIA NIM
microservices for ease of deployment.
Acknowledgments
We would like to thank Liwei Ma, Fanrong Li, Nikita Korobov, and Martin Marciniszyn Mehringer  for their efforts in supporting this post.
使用 NVIDIA TensorRT-LLM 前瞻性解码优化 Qwen2.5-Coder 吞吐量
专注于编码的
大语言模型（LLMs）
已稳步应用于开发者工作流程。从配对编程到自我改进的
AI 智能体
，这些模型可帮助开发者完成各种任务，包括增强代码、修复错误、生成测试和编写文档。
为促进开源 LLM 的开发，Qwen 团队最近发布了 Qwen2.5-Coder，这是一系列先进的 LLM，用于跨热门编程语言的代码生成、推理和修复。本文将探讨针对
NVIDIA TensorRT-LLM 支持
的 Qwen2.5-Coder 模型进行推理优化的优势，以及借助 NVIDIA NIM 轻松部署以提升变革潜力和编码效率的好处。
Qwen2.5-Coder 模型
Qwen2.5-Coder 模型在热门的学术基准测试中取得了出色的性能。
NVIDIA TensorRT-LLM
已对 Qwen2.5-Coder 系列的三种热门模型 (1.5B、7B 和 32B 版本) 进行优化，以实现高吞吐量和低延迟。TensorRT-LLM 是一个用于快速、高效 LLM 推理的库，包含动态机上
批处理
、
KV 缓存
、
KV 缓存重复使用
和几种预测性解码技术等优化功能。
这些优化有助于提高 Qwen2.5-Coder 模型在 Python、C++、Java、Bash、Javascript、TypeScript 和 Go 等热门编程语言中的性能，从而使更多开发者受益。本文将探讨 lookahead decoding 优化的前瞻性及其有助于实现的性能提升。开发者无需进行任何额外训练，也无需额外的草图模型，即可利用 TensorRT-LLM 高级 API 加速 Qwen2.5-Coder 推理，以生成多行自动代码完成。
解码前景展望
解码前瞻是一种预测性解码技术，可解决 LLMs 缓慢自回归的问题。每个自回归解码步骤一次仅生成一个 token，无法利用 NVIDIA GPUs 强大的并行处理能力，导致 GPU 利用率低、吞吐量低。我们之前讨论过通过草稿目标预测解码可以实现的吞吐量提升，在这里，我们讨论了以 Qwen2.5-Coder 模型为例，利用 TensorRT-LLM lookahead decoding 实现的优势。
与自回归解码中的单令牌生成不同，前瞻性解码可同时生成多个令牌，充分利用 GPU 的并行处理能力，利用计算（FLOPs）降低延迟。此外，对于草稿目标预测性解码，前瞻性解码不需要使用单独的草稿模型。
每个解码步骤分为两个并行分支，即 lookahead 分支和验证分支。通过使用经典的非线性系统求解器
Jacobi 迭代法
，lookahead 分支通过生成 n-grams 来对未来的 tokens 执行并行解码。验证分支选择并验证由 lookahead 分支生成的有前景的 n-gram 候选项。
前瞻性算法使用三个关键参数进行配置：窗口大小（W），n-gram 大小（N）和验证集大小（G）。
窗口大小 (W)：表示前瞻性窗口大小，它决定了算法在每个步骤中尝试预测的未来令牌数量。窗口大小越大，模型的视野越广，一次传递就能生成更多 token。这可有效提高吞吐量性能，同时高效利用 GPU 计算 FLOPs。
N-gram size (N)：表示前瞻性流程中使用的 N – gram 的大小。例如，5-gram 是由 5 个未来令牌组成的连续序列。它与窗口大小一起为前瞻性分支创建了一个大小固定的 2D 窗口，以便从 Jacobi 迭代轨迹生成 n-gram。
验证集大小 (G)：表示算法在每个验证步骤中考虑的推测或候选 n-gram 的最大数量。它平衡了计算效率与探索更多可能性之间的权衡。
图 1、使用 (W，N，G) = (5，3，2) 展望解码工作流程。图片来源：
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
未来的性能很大程度上取决于基础模型、硬件、批量大小、序列长度和数据集。建议分析各种配置，以找到给定设置的最佳 (W，N，G) 配置。最佳 (W，N，G) 元组配置支持 lookahead 解码前瞻性，无需任何其他训练、fine-tuning 或 draft 模型，即可提供更高的吞吐量性能。
通过对 (W，N，G) 配置值扫描的实验，我们分别为 Qwen2.5-Coder 7B Instruct 和 Qwen2.5-Coder 32B Instruct 模型实现了 3.6 倍和 1.6 倍的吞吐量加速。这些加速是通过
NVIDIA H100 Tensor Core GPUs
上的吞吐量 (tokens/second) 与基线 (无 lookahead speculative decoding) 的比较进行测量的，如 Figure 2 所示。
图 2、借助 TensorRT-LLM 超前解码，Qwen2.5-Coder 模型可提升 NVIDIA DGX H100 上的吞吐量
数据测量日期：2025 年 1 月 30 日。Qwen2.5-Coder 7B Instruct 和 Qwen2.5-Coder 32B Instruct 模型的推理吞吐量（输出令牌/秒）加速。DGX H100，TP=1 | （W，N，G）= （8，8，8）| Qwen2.5-Coder 7B Instruct，TP=2 | （W，N，G）= （15，15，15）| Qwen2.5-Coder-32B-Instruct，批量大小=1，TensorRT-LLM 版本 0.15.0。
NVIDIA H200 Tensor Core GPU
也实现了类似的吞吐量加速。凭借更高的显存带宽，它们还有助于提高基准吞吐量性能，从而使速度略低于 H100 GPU (图 3)。
图 3、Qwen2.5-Coder 模型在 NVIDIA DGX H200 上通过 TensorRT-LLM 超前解码实现吞吐量提升
数据测量日期：2025 年 1 月 30 日。Qwen2.5-Coder 7B Instruct 和 Qwen2.5-Coder 32B Instruct 模型的推理吞吐量（输出令牌/秒）加速。DGX H200，TP=1 | （W，N，G）= （8，8，8）| Qwen2.5-Coder 7B Instruct，TP=2 | （W，N，G）= （15，15，15）| Qwen2.5-Coder 32B Instruct，批量大小=1，TensorRT-LLM 版本 0.15.0。
使用 TensorRT-LLM 进行解码的前瞻性运行步骤
要在 TensorRT-LLM 中使用预测性解码重现这些性能提升，请执行以下步骤。
# Install TensorRT-LLM. (Commands below are for Linux. Refer to TensorRT-LLM docs for Windows)

sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade setuptools 
&& pip3 install tensorrt_llm --extra-index-url https://pypi.nvidia.com
然后，使用高级 API 在 TensorRT-LLM 中运行 lookahead decoding。
# Command for Qwen2.5-Coder-7B-Instruct

from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi import (LLM, BuildConfig, KvCacheConfig, 
LookaheadDecodingConfig, SamplingParams)
def main():
    """The end user can customize the build configuration with the 
build_config class. # Max draft length is based on (W,N,G) values and 
calculated as: (W + G -1) * (N-1) + ( N<=1 ? 0: N-2)"""

    build_config = BuildConfig(max_batch_size = 128,
max_input_len = 2048, 
max_seq_len = 4096,
max_num_tokens = 16384, 
max_draft_len = 111)
    build_config.plugin_config.reduce_fusion = True
    build_config.plugin_config.use_paged_context_fmha = True 
    build_config.plugin_config.multiple_profiles = True
    
    # The configuration for lookahead decoding
    lookahead_config = LookaheadDecodingConfig(max_window_size=8,
                                               max_ngram_size=8,
                                              max_verification_set_size=8)
    
    kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.4)
    llm = LLM(model="Qwen/Qwen2.5-Coder-7B-Instruct",
              kv_cache_config=kv_cache_config,
              build_config=build_config,
              speculative_config=lookahead_config)
    
    prompt = """Write a C++ program to find the nth Fibonacci number using 
recursion. Now we define a sequence of numbers in which each number is the 
sum of the three preceding ones. The first three numbers are 0, -1, -1. 
Write a program to find the nth number."""
    
    sampling_params = SamplingParams(lookahead_config=lookahead_config)
    
    output = llm.generate(prompt, sampling_params=sampling_params)
    print(output)
if __name__ == '__main__':
    main()
总结
前瞻性预测解码可提高 LLMs 的吞吐量，而无需任何其他训练、微调或草稿模型。我们展示了 Qwen2.5-Coder 模型的基准性能改进。
访问 build.nvidia.com，免费试用通过 NVIDIA TensorRT-LLM 优化的 Qwen2.5-Coder 模型。
为便于部署， 我们还将通过 TensorRT-LLM 优化的 Qwen2.5-Coder 模型打包为可下载的
NVIDIA NIM
微服务。
致谢
在此，
我们要感谢马立伟、李凡融、Nikita Korobov 和 Martin Marciniszyn Mehringer 为支持这篇博文所付出的努力
。