Optimize AI Inference Performance with NVIDIA Full-Stack Solutions
The explosion of AI-driven applications has placed unprecedented demands on both developers, who must balance delivering cutting-edge performance with managing operational complexity and cost, and AI infrastructure.
NVIDIA is empowering developers with full-stack innovations—spanning chips, systems, and software—that redefine what’s possible in
AI inference
, making it faster, more efficient, and more scalable than ever before.
Easily deploy high-throughput, low-latency inference
Six years ago, NVIDIA set out to create an AI inference server specifically designed for developers building high-throughput, latency-critical production applications. At the time, many developers were grappling with custom, framework-specific servers that increased complexity, drove up operational costs, and struggled to meet stringent service-level agreements for latency and throughput.
To address this, NVIDIA developed the
NVIDIA Triton Inference Server
, an open-source platform capable of serving models from any AI framework. By consolidating framework-specific inference servers, Triton streamlined AI inference deployment and increased AI prediction capacity. This approach has made Triton one of the most widely adopted NVIDIA
open-source projects
, now used by
hundreds of leading organizations to deploy production AI models efficiently.
In addition to Triton, NVIDIA offers a broad ecosystem of AI inference solutions. For developers seeking powerful, customizable tools,
NVIDIA TensorRT
provides a high-performance deep learning inference library with APIs that enable fine-grained optimizations.
NVIDIA NIM
microservices provide a flexible framework for deploying AI models across the cloud, data centers, or workstations.
Optimizations for AI inference workloads
Inference is a full-stack problem today, requiring high-performance infrastructure and efficient software to make effective use of that infrastructure. In addition, inference workloads continue to become more challenging, as model sizes continue to grow and latency constraints tighten, all while the number of users leveraging these AI services also continues to increase. And with the introduction of inference time scaling, a new paradigm for scaling model intelligence, more compute is being applied during inference to enhance model performance.
These trends mean that it’s important to continue advancing delivered inference performance, even on the same underlying hardware platform. By combining established methods like model parallelism, mixed-precision training, pruning, quantization, and data preprocessing optimization with cutting-edge advancements in inference technologies, developers can achieve remarkable gains in speed, scalability, and cost-effectiveness.
The TensorRT-LLM library incorporates many state-of-the-art features that accelerate inference performance for
large language models (LLMs)
, which are outlined below.
Prefill and KV cache optimizations
Key-value (KV) cache early reuse
: By reusing system prompts across users, the KV Cache Early Reuse feature accelerates time-to-first-token (TTFT) by up to 5x. Flexible KV block sizing and efficient eviction protocols ensure seamless memory management, enabling faster response times even in multi-user environments.
Chunked prefill
: For smarter deployment, chunked prefill divides the prefill phase into smaller tasks, enhancing GPU utilization and reducing latency. This innovation simplifies deployment and ensures consistent performance, even with fluctuating user demands.
Supercharging multiturn interactions
:
The NVIDIA GH200 Superchip architecture enables efficient KV cache offloading, improving TTFT by up to 2x in multiturn interactions with Llama models while maintaining high throughput.
Decoding optimization
Multiblock attention for long sequences
: Addressing the challenge of long input sequences, TensorRT-LLM multiblock attention maximizes GPU utilization by distributing tasks across streaming multiprocessors (SMs). This technique improves system throughput by more than 3x, enabling support for larger context lengths without additional hardware costs.
Speculative decoding for accelerated throughput
: Leveraging a smaller draft model alongside a larger target model, speculative decoding enables up to a 3.6x improvement in inference throughput. This approach ensures high-speed, high-accuracy generation of model outputs, streamlining workflows for large-scale AI applications.
Speculative decoding with Medusa:
The Medusa speculative decoding algorithm is available as part of TensorRT-LLM optimizations. By predicting multiple subsequent tokens simultaneously, Medusa boosts throughput for Llama 3.1 models by up to 1.9x on the NVIDIA HGX H200 platform. This innovation enables faster responses for applications that rely on LLMs, such as customer support and content creation.
Multi-GPU inference
MultiShot communication protocol
: Traditional Ring AllReduce operations can become a bottleneck in multi-GPU scenarios. TensorRT-LLM MultiShot, powered by
NVSwitch
, reduces communication steps to just two, irrespective of GPU count. This innovation boosts AllReduce speeds by up to 3x, making low-latency inference scalable and efficient.
Pipeline parallelism for high-concurrency efficiency
:
Parallelism techniques require that GPUs be able to transfer data quickly and efficiently, necessitating a robust GPU-to-GPU interconnect fabric for maximum performance. Pipeline parallelism on
NVIDIA H200 Tensor Core
GPUs achieved a 1.5x throughput increase for Llama 3.1 405B and demonstrated their versatility with a 1.2x speedup for Llama 2 70B in MLPerf Inference benchmarks. MLPerf Inference is a suite of industry-standard inference performance benchmarks developed by the MLCommons consortium.
Large NVLink domains:
The NVIDIA GH200 NVL32 system, powered by 32 NVIDIA GH200 Grace Hopper Superchips connected using the NVLink Switch system, and with TensorRT-LLM improvements, delivers up to 3x faster TTFT for Llama models. With up to 127 petaflops of AI compute, this next-generation architecture sets the stage for unprecedented real-time responsiveness in AI applications.
Quantization and lower-precision compute
NVIDIA TensorRT Model Optimizer for precision and performance:
The NVIDIA custom FP8 quantization recipe in the
NVIDIA TensorRT Model Optimizer
delivers up to 1.44x higher throughput without sacrificing accuracy. These optimizations enable more cost-effective deployment by reducing latency and hardware requirements for demanding workloads.
End-to-end full-stack optimization:
NVIDIA TensorRT libraries and FP8 Tensor Core innovations ensure high performance across a wide range of devices, from data center GPUs to edge systems. NVIDIA has optimized the Llama 3.2 collection of models for great performance, demonstrating how full-stack software can adaptively unlock efficiency across diverse AI deployment environments.
With these features, as well as many others within Triton and TensorRT-LLM, developers can now deploy LLMs that are not only faster and more efficient but also capable of handling a wider range of tasks and user demands. This opens new opportunities for businesses to enhance customer service, automate complex processes, and gain deeper insights from their data.
Evaluating inference performance
Delivering world-class inference performance takes a full technology stack—chips, systems, and software—all contributing to boosting throughput, reducing energy consumption per token, and minimizing costs.
MLPerf Inference is one key measure of inference performance is MLPerf Inference. The benchmark measures inference throughput under standardized conditions, with results subject to extensive peer review. The benchmark is regularly updated to reflect new advances in AI, ensuring that organizations can rely on these results to evaluate platform performance.
In the latest round of MLPerf Inference,
NVIDIA Blackwell made its debut
, delivering up to 4x more performance than the NVIDIA H100 Tensor Core GPU on the Llama 2 70B benchmark. This achievement was the result of the many architectural innovations at the heart of the Blackwell GPU, including the second-generation Transformer Engine with FP4 Tensor Cores and ultrafast HBM3e GPU memory that delivers 8 TB/s of memory bandwidth per GPU.
In addition, many aspects of the NVIDIA software stack, including NVIDIA TensorRT-LLM, were re-engineered to make use of new capabilities in Blackwell, such as support for FP4 precision, while continuing to meet the rigorous accuracy target of the benchmark.
The NVIDIA H200 Tensor Core GPU, available now from server makers and cloud service providers, also achieved outstanding results on every benchmark in the data center category. This includes the newly added Mixtral 8x7B mixture-of-experts (MoE) LLM, as well as on the Llama 2 70B LLM and Stable Diffusion XL text-to-image tests. As a result of continued software improvements, the Hopper architecture delivered up to 27% more inference performance compared to the prior round.
NVIDIA Triton Inference Server, running on a system with eight H200 GPUs
achieved virtually identical performance compared to the
NVIDIA bare-metal submission on the Llama 2 70B benchmark
in MLPerf Inference v4.1. This shows that enterprises no longer need to choose between a feature-rich, production-grade AI inference server and peak throughput performance—both can be achieved simultaneously with NVIDIA Triton.
The future of AI inference: Emerging trends and technologies
The landscape of AI inference is rapidly evolving, driven by a series of groundbreaking advancements and emerging technologies. Models continue to get smarter, as increases in compute at data center scale enable pretraining larger models. The introduction of sparse mixture-of-experts model architectures, such as GPT-MoE 1.8T, will also help boost model intelligence while improving compute efficiency. These larger models, whether dense or sparse, will require that GPUs individually become much more capable. NVIDIA Blackwell architecture is set to fuel next-generation generative AI inference.
Each Blackwell GPU features second-generation Transformer Engine and fifth-generationTensor Cores utilizing FP4. Lower-precision data formats help to increase computational throughput and reduce memory requirements. To ensure they can deliver significant performance benefits while maintaining high accuracy, an incredible amount of software craftsmanship is needed.
At the same time, to serve the most demanding models at brisk, real-time rates, many of the most capable GPUs will need to work in concert to generate responses.
The
NVIDIA GB200 NVL72
rack-scale solution creates a 72-GPU NVLink domain that acts as a single massive GPU. For GPT-MoE 1.8T real-time inference, it provides up to a 30x improvement in throughput compared to the prior generation Hopper GPU.
In addition, the emergence of a new scaling law—test-time compute—is providing yet another way to improve response quality and accuracy for even more complex tasks. This new paradigm, first introduced with the OpenAI o1 model, enables models to “reason” by generating many intermediate tokens before outputting the final result. Reasoning models are particularly helpful in domains such as complex mathematics and generating computer code. This new paradigm is set to fuel a new wave of breakthroughs requiring more computational performance during inference time.
The path to artificial general intelligence will rely on continued breakthroughs in data center compute performance. Pretraining, post-training, and test-time scaling all depend on state-of-the-art infrastructure running expertly crafted software. The NVIDIA platform is evolving rapidly, with a brisk one-year innovation rhythm, to enable the ecosystem to continue pushing the frontiers of AI.
Get started
Check out
How to Get Started with AI Inference
, learn more about the
NVIDIA AI Inference
platform, and stay informed about the latest
AI inference performance
updates.
Watch a demo on
how to quickly deploy NVIDIA NIM microservices
or read
A Simple Guide to Deploying Generative AI with NVIDIA NIM
. Optimizations from TensorRT, TensorRT-LLM, and TensorRT Model Optimizer libraries are combined and available through production-ready deployments using
NVIDIA NIM
microservices.
借助 NVIDIA 全栈解决方案提升 AI 推理性能
AI 驱动的应用的爆炸式发展对开发者提出了前所未有的要求，他们必须在提供先进的性能与管理运营复杂性和成本以及 AI 基础设施之间取得平衡。
NVIDIA 正在为开发者提供涵盖芯片、系统和软件的全栈创新，重新定义
AI 推理
的可能性，使其比以往更快、更高效、更具可扩展性。
轻松部署高吞吐量、低延迟推理
六年前，NVIDIA 着手打造 AI 推理服务器，专为构建高吞吐量、延迟关键型生产应用的开发者而设计。当时，许多开发者都在努力使用定制的、特定于框架的服务器，这些服务器增加了复杂性，增加了运营成本，并且难以满足严格的服务水平协议（service-level agreements）关于延迟和吞吐量的要求。
为解决这一问题，NVIDIA 开发了
NVIDIA Triton Inference Server
，这是一个开源平台，能够为来自任何 AI 框架的模型提供服务。通过整合特定于框架的推理服务器，Triton 简化了 AI 推理部署，并提高了 AI 预测能力。这种方法使 Triton 成为广泛采用的 NVIDIA 开源项目之一，现已被数百家领先的组织用于高效部署生产级 AI 模型。
除 Triton 外，NVIDIA 还提供广泛的 AI 推理解决方案生态系统。对于寻求功能强大的可定制工具的开发者，
NVIDIA TensorRT
提供了一个高性能深度学习推理库，其 API 可实现细粒度优化。
NVIDIA NIM
微服务提供了一个灵活的框架，用于在云端、数据中心或工作站中部署 AI 模型。
针对 AI 推理工作负载进行优化
推理是当今的全栈问题，需要高性能基础架构和高效软件来有效利用该基础架构。此外，随着模型大小不断增长和延迟限制日益严格，推理工作负载的挑战性也越来越高，同时利用这些 AI 服务的用户数量也在不断增加。随着推理时间扩展（一种扩展模型智能的新范式）的引入，推理过程中应用了更多的计算来增强模型性能。
这些趋势意味着，即使在相同的底层硬件平台上，继续提高交付的推理性能也很重要。通过将模型并行、混合精度训练、剪枝、量化和数据预处理优化等成熟方法与推理技术的前沿进步相结合，开发者可以在速度、可扩展性和成本效益方面实现显著提升。
TensorRT-LLM 库包含许多先进功能，可加速
大语言模型（LLMs）
的推理性能，如下所述。
预填充和 KV 缓存优化
键值 (KV) 缓存提早复用
：通过在不同用户中重复使用系统提示，KV 缓存提早复用功能可将首个令牌 (TTFT) 的时间缩短高达 5 倍。灵活的 KV 块大小和高效的驱逐协议可确保无缝管理内存，即使在多用户环境中也能缩短响应时间。
分块预填充
：为实现更智能的部署，分块预填充可将预填充阶段划分为较小的任务，从而提高 GPU 利用率并降低延迟。这项创新可简化部署，并确保一致的性能，即使在用户需求波动的情况下也是如此。
强效助力多圈交互
：NVIDIA GH200 超级芯片架构可实现高效的 KV 缓存卸载，在与 Llama 模型进行多圈交互时，将 TTFT 性能提升高达 2 倍，同时保持高吞吐量。
解码优化
长序列的
多块注意力
：TensorRT-LLM 多块注意力通过在流多处理器 (SM) 中分配任务，更大限度地提高 GPU 利用率，从而解决长输入序列的挑战。此技术可将系统吞吐量提高 3 倍以上，从而在不增加硬件成本的情况下支持更大的上下文长度。
用于加速吞吐量的推理吞吐量：通过利用较小的草稿模型和较大的目标模型，推理吞吐量可将推理吞吐量提升高达 3.6 倍。这种方法可确保高速、高精度地生成模型输出，简化大规模 AI 应用的工作流。
使用 Medusa 进行推理解码
：Medusa 推理解码算法可作为 TensorRT-LLM 优化的一部分提供。通过同时预测多个后续令牌，Medusa 在 NVIDIA HGX H200 平台上将 Llama 3.1 模型的吞吐量提高了 1.9 倍。这项创新可加快客户支持和内容创建等依赖 LLM 的应用的响应速度。
多 GPU 推理
MultiShot 通信协议
：传统的 Ring AllReduce 操作可能会成为多 GPU 场景中的瓶颈。TensorRT-LLM MultiShot 由
NVSwitch
提供支持，无论 GPU 数量如何，都可以将通信步骤减少到两个。这项创新将 AllReduce 速度提升高达 3 倍，使低延迟推理具有可扩展性并十分高效。
实现高并发效率的工作流并行：并行技术要求 GPU 能够快速高效地传输数据，因此需要强大的 GPU 到 GPU 互连结构来实现出色性能。
NVIDIA H200 Tensor Core
GPU 上的工作流并行将 Llama 3.1 405B 的吞吐量提高了 1.5 倍，并在 MLPerf Inference 基准测试中证明了其通用性，将 Llama 2 70B 的速度提高了 1.2 倍。MLPerf Inference 是一套行业标准推理性能基准测试，由 MLCommons 联盟开发。
大型 NVLink 域
：NVIDIA GH200 NVL32 系统由通过 NVLink Switch 系统连接的 32 个 NVIDIA GH200 Grace Hopper 超级芯片提供支持，并进行了 TensorRT-LLM 改进，可为 Llama 模型提供高达 3 倍的 TTFT 速度。凭借高达 127 Petaflops 的 AI 计算能力，此新一代架构为 AI 应用实现出色的实时响应速度奠定了基础。
量化和低精度计算
用于提高精度和性能的 NVIDIA TensorRT 模型优化器
：NVIDIA TensorRT 模型优化器中的 NVIDIA 定制 FP8 量化方法可在不牺牲准确性的情况下将吞吐量提高 1.44 倍。这些优化可降低高要求工作负载的延迟和硬件需求，从而实现更具成本效益的部署。
端到端全栈优化
：NVIDIA TensorRT 库和 FP8 Tensor Core 创新技术可确保从数据中心 GPU 到边缘系统等各种设备实现高性能。NVIDIA 优化了 Llama 3.2 模型集合，以实现出色性能，展示了全栈软件如何在不同的 AI 部署环境中灵活释放效率。
借助这些功能以及 Triton 和 TensorRT-LLM 中的许多其他功能，开发者现在可以部署更快速、更高效的 LLM，并且能够处理更广泛的任务和用户需求。这为企业增强客户服务、实现复杂流程自动化以及从数据中获得更深入见解带来了新机遇。
评估推理性能
实现出色的推理性能需要完整的技术堆栈（芯片、系统和软件），所有这些都有助于提高吞吐量、降低每个令牌的能耗并更大限度地降低成本。
MLPerf Inference 是衡量推理性能的一个关键指标。该基准测试用于测量标准化条件下的推理吞吐量，并对结果进行广泛的同行评审。基准测试会定期更新，以反映 AI 领域的新进展，确保企业组织可以依靠这些结果来评估平台性能。
在最新一轮 MLPerf Inference 中，
NVIDIA Blackwell 首次亮相
，在 Llama 2 70B 基准测试中，其性能比 NVIDIA H100 Tensor Core GPU 高 4 倍。这一成就得益于 Blackwell GPU 核心的众多架构创新，包括采用 FP4 Tensor Cores 的第二代 Transformer Engine 和可为每个 GPU 提供 8 TB/s 的 HBM3e GPU 内存带宽。
此外，对 NVIDIA 软件堆栈的许多方面 (包括 NVIDIA TensorRT-LLM) 进行了重新设计，以利用 Blackwell 中的新功能 (例如对 FP4 精度的支持)，同时继续满足基准测试的严格准确性目标。
服务器制造商和云服务提供商现已推出的 NVIDIA H200 Tensor Core GPU 在数据中心类别的每项基准测试中都取得了出色的成绩。其中包括新增的 Mixtral 8x7B 多专家模型 (MoE) LLM，以及 Llama 2 70B LLM 和 Stable Diffusion XL 文本转图像测试。得益于软件的持续改进，Hopper 架构可提供高达 27% 的推理性能。
与 MLPerf Inference v4.1 中 Llama 2 70B 基准测试中的 NVIDIA 裸机提交相比
，在配备 8 个 H200 GPU 的系统上运行的 NVIDIA Triton Inference Server 实现了几乎相同的性能。这表明企业不再需要在功能丰富的生产级 AI 推理服务器和峰值吞吐量性能之间做出选择，而
NVIDIA Triton 可以同时实现这两种性能。
AI 推理的未来：新兴趋势和技术
在一系列突破性进展和新兴技术的推动下，AI 推理的格局正在迅速发展。随着数据中心规模的计算能力增加，模型将继续变得更加智能。引入稀疏的多专家模型架构 (例如 GPT-MoE 1.8T) 也将有助于提高模型智能，同时提高计算效率。这些更大型的模型，无论是密集模型还是稀疏模型，都需要 GPU 单独变得更加强大。NVIDIA Blackwell 架构将为新一代生成式 AI 推理提供动力支持。
每个 Blackwell GPU 均配备第二代 Transformer Engine 和第五代 Tensor Cores，利用 FP4。低精度数据格式有助于提高计算吞吐量并降低内存需求。为了确保它们能够在保持高精度的同时提供显著的性能优势，我们需要大量的软件技术。
与此同时，为了以快速、实时的速率为要求严苛的模型提供服务，许多功能非常强大的 GPU 需要协同工作以生成响应。
NVIDIA GB200 NVL72
机架级解决方案创建了一个 72-GPU NVLink 域，可充当单个大型 GPU。对于 GPT-MoE 1.8T 实时推理，与上一代 Hopper GPU 相比，其吞吐量提高了 30 倍。
此外，新的扩展定律（测试时计算） 的出现为提高更复杂任务的响应质量和准确性提供了另一种方法。这种新范式首先在 OpenAI o1 模型中引入，使模型能够在输出最终结果之前通过生成许多中间令牌来“推理”。推理模型在复杂数学和生成计算机代码等领域尤为有用。这种新范式将起新一轮突破浪潮，需要在推理期间实现更高的计算性能。
通往人工通用智能的道路将依赖于数据中心计算性能的持续突破。预训练、后训练和测试时扩展都依赖于运行专家精心编写的软件的最先进的基础架构。NVIDIA 平台发展迅速，一年内创新节奏轻快，使生态系统能够继续推动人工智能的前沿发展。
开始使用
查看如何开始使用 AI 推理
，了解更多关于
NVIDIA AI 推理平台
的信息，并随时了解
最新的 AI 推理性能更新
。
观看演示，了解如何快速部署
NVIDIA NIM
微服务，或阅读《使用 NVIDIA NIM 部署生成式 AI 的简单指南》。TensorRT、TensorRT-LLM 和 TensorRT Model Optimizer 库中的优化经过组合，可通过使用 NVIDIA NIM 微服务的生产就绪型部署获得。