Boost Llama 3.3 70B Inference Throughput 3x with NVIDIA TensorRT-LLM Speculative Decoding
Meta’s Llama collection of open
large language models (LLMs)
continues to grow with the recent addition of
Llama 3.3 70B
, a text-only instruction-tuned model. Llama 3.3 provides enhanced performance respective to the older Llama 3.1 70B model and can even match the capabilities of the larger, more computationally expensive
Llama 3.1 405B model
on several tasks including math, reasoning, coding, and multilingual support.
NVIDIA TensorRT-LLM
, a powerful inference engine that delivers state-of-the-art performance on the latest LLMs, incorporates many optimizations to deliver outstanding  Llama 3.3 70B inference throughput. These include
in-flight batching
,
KV caching
,
custom FP8 quantization
, speculative decoding, and more for fast, cost-efficient LLM serving.
With in-flight batching activated by default as a runtime configuration parameter, TensorRT-LLM supports batching multiple different requests at the same time for higher serving throughput. By interleaving requests in context and generation phases, in-flight batching reduces latency and improves GPU utilization by executing new requests while older requests are still in flight. Finished requests are evicted from the batch, making room for the next set of requests.
Caching the values of the key-value elements of previous tokens saves from expensive recomputation of these tensors in the generation phase for the next set of tokens. Computational savings effectively lead to higher throughput. However, KV cache grows linearly in size with number of batched requests and sequence context lengths, leading to higher memory requirements.
TensorRT-LLM KV caching addresses these challenges through several optimizations, including support for
paged KV cache
,
quantized KV cache
,
circular buffer KV cache
and
KV cache reuse
. Each of these optimizations address the challenging balance between growing memory size and avoiding unnecessary and expensive recomputation.Speculative decoding is a popular technique for faster and cost-effective LLM inference with built-in verification for the quality of output generation. It’s  based on the premise that generating multiple sequences of future (draft) tokens is more efficient than processing a single token in autoregressive decoding, an inherently time-consuming process. The target model determines how many of these draft tokens to accept, which is far more efficient than having to generate one token per iteration. TensorRT-LLM supports a growing list of speculative decoding techniques including
draft target
,
Medusa
,
Eagle
, and
lookahead decoding
, among others.
Figure 1. NVIDIA TensorRT optimizations for high-performance deep learning inference
In this post, we show how the
NVIDIA HGX H200 platform
with NVLink and NVSwitch, as well as TensorRT-LLM, achieve great performance when running the latest Llama 3.3 70B model. We describe the step-by-step setup to get speculating decoding working for Llama 3.3 70B with TensorRT-LLM. For more information, including other optimizations, different models, and multi-GPU execution, see the full list of
TensorRT-LLM examples
.
Achieving throughput speedups with draft target speculative decoding
Table 1 and Figure 2 highlight the throughput (output tokens/second) speedups between no draft model (that is, no speculative decoding) versus draft models of various sizes with Llama 3.3 70B target model.
Throughput Performance – Output Tokens/Second
One NVIDIA H200 Tensor Core GPU
Draft | Target
Models
Llama 3.2 1B | Llama 3.3 70B
Llama 3.2 3B | Llama 3.3 70B
Llama 3.1 8B | Llama 3.3 70B
Llama 3.3 70B
(without draft model)
Tokens/sec
181.74
161.53
134.38
51.14
Speedups (with versus without draft models)
3.55x
3.16x
2.63x
N/A
Table 1. Throughput performance using one NVIDIA H200 Tensor Core GPU with TensorRT-LLM internal measurements
Data measured on December 11, 2024. Output tokens/second is inclusive of time to generate the first token – tok/s = total generated tokens  / total latency. DGX H200, TP1, FP8, batch size=1, TensorRT Model Optimizer version 0.21, TensorRT-LLM version 0.15.0​.
Figure 2. Throughput speedups using speculative decoding with Llama 3.3 70B target model
We provide the steps to reproduce these performance gains using draft target speculative decoding within TensorRT-LLM.
# Download the following model checkpoints from Hugging Face and store them 
in a directory for easy access through the setup process.

git lfs install

# Download target models
git clone https://huggingface.co/meta-llama/Meta-Llama-3.3-70B-Instruct

# Download draft models
git clone https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct
git clone https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
git clone https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
After the model checkpoints have been downloaded, install TensorRT-LLM.
# Obtain and start the basic docker image environment (optional).
docker run --rm --ipc=host --runtime=nvidia --gpus all --entrypoint 
/bin/bash -it nvidia/cuda:12.5.1-devel-ubuntu22.04

# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin 
libopenmpi-dev git git-lfs

# Fetch the library
git clone -b v0.15.0 https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Install the latest version (corresponding to the main branch) of TensorRT-LLM.
pip3 install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com

# Check installation
python3 -c "import tensorrt_llm"
Next, compile the downloaded model checkpoints into draft and target TensorRT engines. These engines are optimized to run inference with best accuracy and highest throughput.
cd examples

# Steps to build target and draft models in FP8 precision on 1 H200

# Create FP8 checkpoints

python3 quantization/quantize.py --model_dir <path to draft model repo> --dtype float16 --qformat fp8 --kv_cache_dtype fp8 
--output_dir /ckpt-draft --calib_size 512 --tp_size 1

python3 quantization/quantize.py \
    --model_dir=<path to target model repo> \
    --output_dir=./ckpt-target-70b \
    --dtype=float16 --qformat fp8 --kv_cache_dtype fp8 \
    --calib_size 512 --tp_size 1 

# Build draft and target engines
# Important flags for the engine build process:
# --use_paged_context_fmha=enable must be specified since we need KVcache reuse for the draft/target model.

# --speculative_decoding_mode=draft_tokens_external and --max_draft_len must be specified for target model.

trtllm-build \
    --checkpoint_dir ./ckpt-draft \
    --output_dir=./draft-engine \
    --gpt_attention_plugin float16 \
    --workers 1 \
    --gemm_plugin=fp8 \
    --use_paged_context_fmha=enable \
    --multiple_profiles enable \
    --max_batch_size=32 \
    --max_seq_len=131072

trtllm-build \
    --checkpoint_dir=./ckpt-target-70b \
    --output_dir=./target-engine \
    --gpt_attention_plugin float16 \
    --workers 1 \
    --gemm_plugin=fp8 \
    --use_paged_context_fmha=enable \
    --multiple_profiles enable \
    --max_batch_size=32 \
    --max_seq_len=131072 \
    --low_latency_gemm_plugin fp8 \
    --speculative_decoding_mode=draft_tokens_external \
    --max_draft_len 10
Finally, run speculative decoding in TensorRT-LLM.
#Run decoding

# Important flags to set during the run process:
#--draft_engine_dir and --engine_dir must be specified for the draft and target engines.

#--draft_target_model_config is corresponding to the configuration of 
Draft-Target-Model. As an example, [4,[0],[1],False] means draft_len=4, 
device of draft model is GPU0, device of target model is GPU1, and use 
tokens rather than logits to accept.

# Only CPP session (using executor as low-level API) is supported, while 
Python session (--use_py_session) is not supported.

# Run with Llama 3.3 70B target model

mpirun -n 1 --allow-run-as-root python3 ./run.py \
    --tokenizer_dir <path to draft model repo> \
    --draft_engine_dir ./draft-engine \
    --engine_dir ./target-engine \     
    --draft_target_model_config = "[10,[0,1,2,3,4,5,6,7],[0,1,2,3,4,5,6,7], False]" \
    --kv_cache_free_gpu_memory_fraction=0.35 \
    --max_output_len=1024 \
    --kv_cache_enable_block_reuse \
    
--input_text="<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nA 
3-digit integer contains one of each of the digits 1,3 and 5. What is the 
probability that the integer is divisible by 
5.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n"

# Following is the LLM-generated output:

Output [Text 0 Beam 0]: "## Step 1: Determine the total number of 3-digit 
integers that can be formed using the digits 1, 3, and 5.
There are 3! = 6 ways to arrange the digits 1, 3, and 5 to form different 3-digit integers.

## Step 2: Identify the condition for a number to be divisible by 5.
A number is divisible by 5 if its last digit is either 0 or 5.

## Step 3: Determine the number of arrangements where 5 is the last digit.
Since the digit 5 must be the last digit for the number to be divisible by 
5, we fix the last position with 5. The remaining two positions can be 
filled with the digits 1 and 3 in 2! = 2 ways.

## Step 4: Calculate the probability that the integer is divisible by 5.
The probability is the number of favorable outcomes (arrangements where 5 is the last digit) 
divided by the total number of possible outcomes (total arrangements of the digits 1, 3, and 5).

## Step 5: Calculate the probability.
Probability = (Number of favorable outcomes) / (Total number of outcomes) = 2 / 6 = 1/3.

The final answer is: $\boxed{\frac{1}{3}}$"
To benchmark throughput performance without speculative decoding, follow the steps below:
# Run throughput benchmark for the 70B model without the draft model

trtllm-build --checkpoint_dir ./ckpt-target-70b --output_dir /data/70B-TRT/ 
--gpt_attention_plugin float16 --workers 1 --max_batch_size 32 
--max_seq_len 131072 --use_fused_mlp enable --reduce_fusion enable 
--use_paged_context_fmha enable --multiple_profiles enable --gemm_plugin fp8

python3 /app/tensorrt_llm/benchmarks/cpp/prepare_dataset.py --output 
token-norm-dist.json --tokenizer /llama-3_3-70b/ token-norm-dist 
--num-requests 1000 --input-mean 500 --input-stdev 0 --output-mean 200 
--output-stdev 0 > /tmp/synthetic.txt

trtllm-bench --model <path to target model repo> latency --engine_dir 
/data/70b-TRT/ --dataset /tmp/synthetic.txt
Summary
NVIDIA collaborates with Meta for the creation, optimization, and acceleration of the world’s leading open models. NVIDIA supports Llama as part of our commitment to grow open community AI models and software to enable users to customize and address their own unique workloads. NVIDIA is involved with several open-source projects through partnering with developers, maintainers, and foundations.
NVIDIA TensorRT-LLM
provides several features for optimizing and efficiently running LLMs of different model architectures. These optimizations lead to significant speedups on the same hardware, enable fewer resources to serve the same workload, reduce energy costs, and improve total cost of ownership. Available through production-ready deployments using
NVIDIA NIM
microservices, these TensorRT optimizations accelerate the deployment of your generative AI applications across NVIDIA-accelerated infrastructure anywhere, including cloud, data center, and workstations.
借助 NVIDIA TensorRT-LLM 预测解码，将 Llama 3.3 的推理吞吐量提升 3 倍
随着近期新增的 Llama 3.3 70B
(一种纯文本指令调整模型)，Meta 的开放
大语言模型 (LLMs)
集合将继续增长。Llama 3.3 相对于较旧的 Llama 3.1 70B 模型提供了增强的性能，甚至可以在数学、推理、编码和多语种支持等多项任务中匹配更大、计算成本更高的 Llama 3.1 405B 模型的功能。
NVIDIA TensorRT-LLM
是一款功能强大的推理引擎，可在最新的 LLM 上提供先进的性能，并整合了许多优化措施，可提供出色的 Llama 3.3 70B 推理吞吐量。其中包括在飞批处理、
KV 缓存
、
自定义 FP8 量化
、推测解码等，可实现快速、经济高效的 LLM 服务。
动态批处理默认作为运行时配置参数激活，TensorRT-LLM 支持同时批处理多个不同的请求，从而提高服务吞吐量。通过在上下文和生成阶段交错处理请求，动态批处理可在旧请求仍在运行时执行新请求，从而减少延迟并提高 GPU 利用率。已完成的请求将从批处理中移除，从而为下一组请求腾出空间。
缓存先前令牌的键值元素的值可避免在为下一组令牌生成阶段对这些张量进行昂贵的重新计算。节省的计算量可有效提高吞吐量。但是，随着批量请求的数量和序列上下文长度的增加，KV 缓存的大小呈线性增长，从而导致内存需求增加。
TensorRT-LLM KV 缓存通过多种优化解决了这些挑战，包括对分页 KV 缓存、
量化 KV 缓存
、
循环缓冲区 KV 缓存
和
KV 缓存重复使用
的支持。每项优化都解决了在增大内存大小与避免不必要且昂贵的重新计算之间实现富有挑战性的平衡这一问题。预测解码是一种热门技术，可通过内置验证来生成输出的质量，从而实现更快且经济高效的 LLM 推理。其前提是，在自回归解码过程中，生成多个未来 (草稿) 令牌序列比处理单个令牌更高效。目标模型决定接受这些草稿令牌的数量，这比每次迭代生成一个令牌的效率要高得多。TensorRT-LLM 支持越来越多的预测性解码技术，包括
草稿目标
、
Medusa
、
Eagle
和前瞻性解码等。
图 1、适用于高性能深度学习推理的 NVIDIA TensorRT 优化
在本文中，我们将展示搭载 NVLink 和 NVSwitch 的
NVIDIA HGX H200 平台以及 TensorRT-LLM
如何在运行最新的 Llama 3.3 70B 模型时实现出色性能。我们介绍了使用 TensorRT-LLM 对 Llama 3.3 70B 进行解码预测的分步设置。有关其他优化、不同模型和多 GPU 执行的更多信息，请参阅
TensorRT-LLM 示例
的完整列表。
通过草稿目标预测解码实现吞吐量加速
表 1 和图 2 突出显示了无草稿模型 (即无预测解码) 与使用 Llama 3.3 70B 目标模型的各种规模的草稿模型之间的吞吐量 (输出令牌/秒) 加速。
吞吐量性能 – 输出 Tokens/秒
One NVIDIA H200 Tensor Core GPU
草稿|目标
模型
Llama 3.2 1B|Llama 3.3 70B
Llama 3.2 3B|Llama 3.3 70B
Llama 3.1 8B|Llama 3.3 70B
Llama 3.3 70B
(无草稿模型)
令牌/秒
191.74
151.53
134.38
51.14
加速 (有与无草稿模型对比)
3.55 倍
3.16 倍
2.63 倍
不适用
表 1、使用一个 NVIDIA H200 Tensor Core GPU 和 TensorRT-LLM 内部测量的吞吐量性能
数据测量于 2024 年 12 月 11 日。输出 tokens/second 包含生成第一个 token 的时间 – tok/s = 总生成 tokens / 总延迟。DGX H200、TP1、FP8，批量大小 = 1，TensorRT Model Optimizer 版本 0.21，TensorRT-LLM 版本 0.15.0。
图 2、使用 Llama 3.3 70B 目标模型的预测解码提高吞吐量
我们提供了在 TensorRT-LLM 中使用草稿目标预测解码重现这些性能提升的步骤。
# Download the following model checkpoints from Hugging Face and store them 
in a directory for easy access through the setup process.

git lfs install

# Download target models
git clone https://huggingface.co/meta-llama/Meta-Llama-3.3-70B-Instruct

# Download draft models
git clone https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct
git clone https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
git clone https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
下载模型检查点后，安装 TensorRT-LLM。
# Obtain and start the basic docker image environment (optional).
docker run --rm --ipc=host --runtime=nvidia --gpus all --entrypoint 
/bin/bash -it nvidia/cuda:12.5.1-devel-ubuntu22.04

# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin 
libopenmpi-dev git git-lfs

# Fetch the library
git clone -b v0.15.0 https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Install the latest version (corresponding to the main branch) of TensorRT-LLM.
pip3 install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com

# Check installation
python3 -c "import tensorrt_llm"
接下来，将下载的模型检查点编译到草稿和目标 TensorRT 引擎中。这些引擎经过优化，能够以最佳的准确性和最高的吞吐量运行推理。
cd examples

# Steps to build target and draft models in FP8 precision on 1 H200

# Create FP8 checkpoints

python3 quantization/quantize.py --model_dir <path to draft model repo> --dtype float16 --qformat fp8 --kv_cache_dtype fp8 
--output_dir /ckpt-draft --calib_size 512 --tp_size 1

python3 quantization/quantize.py \
    --model_dir=<path to target model repo> \
    --output_dir=./ckpt-target-70b \
    --dtype=float16 --qformat fp8 --kv_cache_dtype fp8 \
    --calib_size 512 --tp_size 1 

# Build draft and target engines
# Important flags for the engine build process:
# --use_paged_context_fmha=enable must be specified since we need KVcache reuse for the draft/target model.

# --speculative_decoding_mode=draft_tokens_external and --max_draft_len must be specified for target model.

trtllm-build \
    --checkpoint_dir ./ckpt-draft \
    --output_dir=./draft-engine \
    --gpt_attention_plugin float16 \
    --workers 1 \
    --gemm_plugin=fp8 \
    --use_paged_context_fmha=enable \
    --multiple_profiles enable \
    --max_batch_size=32 \
    --max_seq_len=131072

trtllm-build \
    --checkpoint_dir=./ckpt-target-70b \
    --output_dir=./target-engine \
    --gpt_attention_plugin float16 \
    --workers 1 \
    --gemm_plugin=fp8 \
    --use_paged_context_fmha=enable \
    --multiple_profiles enable \
    --max_batch_size=32 \
    --max_seq_len=131072 \
    --low_latency_gemm_plugin fp8 \
    --speculative_decoding_mode=draft_tokens_external \
    --max_draft_len 10
最后，在 TensorRT-LLM 中运行投机性解码。
#Run decoding

# Important flags to set during the run process:
#--draft_engine_dir and --engine_dir must be specified for the draft and target engines.

#--draft_target_model_config is corresponding to the configuration of 
Draft-Target-Model. As an example, [4,[0],[1],False] means draft_len=4, 
device of draft model is GPU0, device of target model is GPU1, and use 
tokens rather than logits to accept.

# Only CPP session (using executor as low-level API) is supported, while 
Python session (--use_py_session) is not supported.

# Run with Llama 3.3 70B target model

mpirun -n 1 --allow-run-as-root python3 ./run.py \
    --tokenizer_dir <path to draft model repo> \
    --draft_engine_dir ./draft-engine \
    --engine_dir ./target-engine \     
    --draft_target_model_config = "[10,[0,1,2,3,4,5,6,7],[0,1,2,3,4,5,6,7], False]" \
    --kv_cache_free_gpu_memory_fraction=0.35 \
    --max_output_len=1024 \
    --kv_cache_enable_block_reuse \
    
--input_text="<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nA 
3-digit integer contains one of each of the digits 1,3 and 5. What is the 
probability that the integer is divisible by 
5.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n"

# Following is the LLM-generated output:

Output [Text 0 Beam 0]: "## Step 1: Determine the total number of 3-digit 
integers that can be formed using the digits 1, 3, and 5.
There are 3! = 6 ways to arrange the digits 1, 3, and 5 to form different 3-digit integers.

## Step 2: Identify the condition for a number to be divisible by 5.
A number is divisible by 5 if its last digit is either 0 or 5.

## Step 3: Determine the number of arrangements where 5 is the last digit.
Since the digit 5 must be the last digit for the number to be divisible by 
5, we fix the last position with 5. The remaining two positions can be 
filled with the digits 1 and 3 in 2! = 2 ways.

## Step 4: Calculate the probability that the integer is divisible by 5.
The probability is the number of favorable outcomes (arrangements where 5 is the last digit) 
divided by the total number of possible outcomes (total arrangements of the digits 1, 3, and 5).

## Step 5: Calculate the probability.
Probability = (Number of favorable outcomes) / (Total number of outcomes) = 2 / 6 = 1/3.

The final answer is: $\boxed{\frac{1}{3}}$"
要在不使用 speculative decoding 的情况下对吞吐量性能进行基准测试，请执行以下步骤：
# Run throughput benchmark for the 70B model without the draft model

trtllm-build --checkpoint_dir ./ckpt-target-70b --output_dir /data/70B-TRT/ 
--gpt_attention_plugin float16 --workers 1 --max_batch_size 32 
--max_seq_len 131072 --use_fused_mlp enable --reduce_fusion enable 
--use_paged_context_fmha enable --multiple_profiles enable --gemm_plugin fp8

python3 /app/tensorrt_llm/benchmarks/cpp/prepare_dataset.py --output 
token-norm-dist.json --tokenizer /llama-3_3-70b/ token-norm-dist 
--num-requests 1000 --input-mean 500 --input-stdev 0 --output-mean 200 
--output-stdev 0 > /tmp/synthetic.txt

trtllm-bench --model <path to target model repo> latency --engine_dir 
/data/70b-TRT/ --dataset /tmp/synthetic.txt
总结
NVIDIA 与 Meta 合作创建、优化和加速世界领先的开放模型。NVIDIA 支持 Llama，履行我们发展开放社区 AI 模型和软件的承诺，使用户能够自定义和处理自己独特的工作负载。NVIDIA 通过与开发者、维护人员和基金会合作参与了多个开源项目。
NVIDIA TensorRT-LLM
提供多种功能，用于优化和高效运行不同模型架构的 LLM。这些优化可显著加快相同硬件的运行速度，减少资源以处理相同工作负载，降低能源成本，并提高总拥有成本。这些 TensorRT 优化通过使用
NVIDIA NIM
微服务的生产就绪型部署提供，可随时随地在 NVIDIA 加速的基础设施 (包括云、数据中心和工作站) 中加速生成式 AI 应用的部署。