NVIDIA DGX Cloud Introduces Ready-To-Use Templates to Benchmark AI Platform Performance
In the rapidly evolving landscape of AI systems and workloads, achieving optimal model training performance extends far beyond chip speed. It requires a comprehensive evaluation of the entire stack, from compute to networking to model framework.
Navigating the complexities of AI system performance can be difficult. There are many application changes that you can make, from precision to parallelism, but they currently require significant effort and specialized knowledge to implement effectively.
NVIDIA DGX Cloud
Benchmarking Recipes offer guidance for boosting training performance by sharing what good performance looks like per workload and best practices for how to get there.
For more information, see
NVIDIA DGX Cloud Benchmarking Recipes
.
Evaluating an AI system holistically
DGX Cloud Benchmarking Recipes are an end-to-end benchmarking suite that can both measure performance in real-world environments and identify optimization opportunities in AI training workloads. These benchmarking recipes are crafted to evaluate performance for real-world AI applications, factoring in the complete AI stack.
Chip-centric metrics, such as peak floating-point operations per second (FLOPS) and bandwidth, can be inadequate in estimating end-to-end performance. Traditionally, peak FLOPS has often been used to compare platforms, but FLOPS is only one of the many components that affect the end-to-end application performance.
In practice, the training time for an AI model is a function of many other components, such as network, software, firmware, and underlying infrastructure.
For example, the high-bandwidth
NVIDIA NVLink
network fabric enables scaling parallelism strategies, such as tensor parallelism, beyond the traditional single-server, 8-GPU limit. With
NVIDIA Grace
class systems, the NVLink networking layer enables you to achieve higher FLOPS in real-world applications, bridging the gap between theoretical and practical performance.
Evaluating AI platforms solely through FLOPS can result in inaccurate estimations of total training time and the associated costs, without accounting for the rest of the platform. For modern AI workloads such as fine-tuning Llama 3.1 models, it’s more accurate to use benchmarks that measure end-to-end performance across the entire system, providing a holistic view of how a platform will perform in actual usage scenarios.
Figure 1. Performance impacts in cloud-based AI development
Infrastructure factors affecting performance include the following:
Server hardware designs
Operating systems
Virtualization layers
Software stacks
Network architectures
Storage implementations
AI workload factors affecting performance include the following:
Compute-to-communication ratio
Model scaling factors
Batch size
Precision format
Data loading strategies
Tuning workloads to optimal performance
Beyond the job execution aspect of benchmarking, NVIDIA DGX Cloud Benchmarking Recipes are also playbooks for optimizing popular models and workloads. These recipes provide workload-specific strategies to maximize performance for popular models such as Llama 3.1, Grok, and Mixtral.
Workload
Type
Description
Container Version
Dataset
Max Scale (#GPUs)
DTYPE
Nemotron4
Training
15B and 340B benchmarks
24.09
Synthetic
2048
FP8, BF16
Nemo Megatron
Training
175B benchmarks
24.05
Pile
2048
FP8, BF16
Llama 3.1
Training
8B, 70B, and 405B benchmarks
24.09
Pile
2304
FP8, BF16
PaXML
Training
5B and 175B benchmarks
24.03.04
Synthetic
2048
FP8, BF16
Maxtext
Training
Llama2 70B benchmarks
2024.12.09
Synthetic
2048
FP8, BF16
Grok1
Training
Grok1 314B benchmarks
24.09
Synthetic
2048
FP8, BF16
Llama 2
Fine Tuning
Hugging Face 70B benchmarks
24.02
HF Llama2
512
BF16
Mistral
Fine Tuning
Hugging Face 7B benchmarks
24.02
HF Mistral
256
BF16
Table 1. Available workloads in the DGX Cloud Benchmarking Recipes catalog
In Table 1, workloads include both training and fine tuning and support both FP8 and BF16 where possible.
Each training workload has a different fingerprint of how it exercises the platform. A basic question you might ask about a workload’s fingerprint is, “How much does compute time overlap with communication or networking time?”
Some models may be more compute-bound and some are more communication bound, depending on the choice of parallelism and hyperparameters such as sequence length and batch size. Scaling behavior also varies between models as the number of GPUs are increased, as well as type of scaling, weak or strong.
For each workload and cluster scale, you must tune your model and system to achieve optimal performance.
On the model side, this may involve adjusting the parallelism strategy, batch sizes, precision formats, and data loading strategies, among other configurations. On the system side, make sure that the workload makes maximal use of NVLink high bandwidth (for example for tensor and context parallelism) and confirm that the scale-out fabric is not a blocker for the corresponding networking collectives (for example for pipeline or expert parallelism).
The latter requires a fabric that provides low transport latency (RDMA) and effective congestion management and adaptive routing as found in the reference
NVIDIA SpectrumX and InfiniBand
networking architectures. For efficient scaling of AI workloads, using these technologies is essential as they help mitigate the impact of jitter, ensuring consistent performance and reliability.
Using FP8
DGX Cloud Benchmarking Recipes provide optimized configurations and tuning recommendations specifically for FP8 workloads, helping you achieve optimal performance with this precision format. For example, the recipe for Llama 3.1 70B training includes FP8 settings that have been carefully tested and optimized for DGX Cloud platforms.
Understanding what constitutes good performance for a given AI workload can be complex. DGX Cloud Benchmarking Recipes provide a range of baseline performance results for various popular models, enabling you to set realistic expectations and goals for your own implementations.
These baselines include metrics such as model FLOPS utilization (MFU), which measures how efficiently a model uses the available compute resources. You can see how MFU and throughput compare for popular models. By comparing your results to these benchmarks, you can gauge the effectiveness of optimizations and identify areas for improvement.
DeepSeek-R1
is a 671B-parameter model that runs on one NVIDIA H200 GPU node. Its high compute utilization shows how holistic optimization of compute, networking, and parallelism strategies can push delivered performance closer to theoretical limits. Systematic benchmarking enables direct comparisons, helping teams collaboratively optimize models and platforms to maximize the value of GPU systems.
Finally, these per-workload performance optimizations also show the need for further research and discussion around application tuning. For example, the recommended usage of the parallelization strategies varies across combinations of workloads and platforms.
Get started with DGX Cloud Benchmarking Recipes
The recipes for benchmarking platform performance are hosted in NVIDIA’s public registry, NGC Catalog. For more information about the latest release of recipes, see
DGX Cloud Benchmarking 24.11.1
.
Within each workload recipe, you can access the following:
Containerized benchmarks for reproducibility across environments
Scripts that generate synthetic data as needed
Performance metrics collection and reporting (to
stdout
)
Configuration best practices for that workload per platform
Performance data from the NVIDIA reference architecture for comparison
The recipes require Slurm cluster management. Support for Kubernetes is currently in development. To use the DGX Cloud Benchmarking Recipes, download the recipe that best matches your workload and execute the cluster setup and benchmarking scripts.
Keep moving the platform performance goalposts
In today’s AI landscape, achieving optimal performance requires looking beyond individual components to understand how entire systems work together. While raw GPU capabilities matter, full optimization comes from carefully tuning every layer of the stack, from hardware and software configuration to workload-specific parameters.
At NVIDIA, we use benchmark recipes to continuously refine every layer of the technology stack, from hardware interconnects such as
NVIDIA NVLink and NVLink Switch
to software libraries such as
NVIDIA TensorRT-LLM,
enabling substantial performance gains over time.
For example, accelerated computing performance has
increased 3.4x in MLPerf Inference on NVIDIA H100 GPUs
in just one year through continuous improvements in software developments alone. These ongoing optimizations enable organizations to run more complex models, reduce infrastructure requirements, and improve efficiency, driving further innovation.
These benchmarking recipes enable your team to:
Optimize AI workloads for specific environments, including for FP8.
Assess how close a cluster’s performance is to NVIDIA’s observed performance.
Identify performance bottlenecks in your current setups.
Training large models can take weeks or months and cost millions in compute resources, so modest performance improvements can translate into substantial time and cost savings. By using continually evolving performance optimizations and workload-specific recipes from NVIDIA, your organization can maximize AI infrastructure investments and focus engineering efforts on innovation rather than infrastructure tuning.
For more information, see
DGX Cloud Benchmarking Recipes
.
NVIDIA DGX 云推出即用型模板，更好地评估 AI 平台性能
在快速发展的 AI 系统和工作负载环境中，实现出色的模型训练性能远远超出芯片速度。这需要对整个堆栈进行全面评估，从计算到网络，再到模型框架。
了解复杂的 AI 系统性能可能比较困难。从精度到并行，您可以对许多应用程序进行更改，但目前需要付出大量努力和掌握专业知识才能有效实施这些更改。
NVIDIA DGX Cloud
Benchmarking Recipes 为提高训练性能提供了指导，通过分享每个工作负载的良好性能以及实现方法的最佳实践。
如需了解更多信息，请参阅
NVIDIA DGX Cloud Benchmarking Recipes
。
全面评估 AI 系统
DGX 云 Benchmarking Recipes 是一款端到端基准测试套件，既可以衡量真实环境中的性能，也可以识别 AI 训练工作负载中的优化机会。这些基准测试方法经过精心设计，可评估真实 AI 应用的性能，并将其纳入完整的 AI 堆栈。
以芯片为中心的指标 (例如每秒浮点运算峰值 (FLOPS) 和带宽峰值) 可能不足以估算端到端性能。传统上，峰值 FLOPS 通常用于比较平台，但 FLOPS 只是影响端到端应用程序性能的众多组件之一。
在实践中，AI 模型的训练时间是许多其他组件（例如网络、软件、固件和底层基础设施）的函数。
例如，高带宽
NVIDIA NVLink
网络结构支持扩展并行策略（例如张量并行），从而超越传统的单服务器 8-GPU 限制。借助
NVIDIA Grace
类系统，NVLink 网络层可让您在实际应用中实现更高的 FLOPS，从而缩小理论性能和实际性能之间的差距。
仅通过 FLOPS 评估 AI 平台会导致对总训练时间和相关成本的估计不准确，而不考虑平台的其余部分。对于现代 AI 工作负载 (例如微调 Llama 3.1 模型)，使用基准来衡量整个系统的端到端性能更为准确，从而全面了解平台在实际使用场景中的表现。
图 1. 基于云的 AI 开发对性能的影响
影响性能的基础架构因素包括：
服务器硬件设计
操作系统
虚拟化层
软件堆栈
网络架构
存储实现
影响性能的 AI 工作负载因素包括：
计算与通信比
模型扩展系数
批量大小
精度格式
数据加载策略
调整工作负载以获得最佳性能
除了基准测试的作业执行方面，NVIDIA DGX Cloud Benchmarking Recipes 也是用于优化热门模型和工作负载的手册。这些方法提供了特定于工作负载的策略，可更大限度地提高 Llama 3.1、Grok 和 Mixtral 等热门模型的性能。
工作负载
类型
说明
容器版本
数据集
最大规模 (GPU 数量)
DTYPE
Nemotron4
训练
150B 和 340B 基准测试
24.09
合成
2048
FP8、BF16
Nemo Megatron
训练
1750 亿次基准测试
24.05
堆栈
2048
FP8、BF16
Llama 3.1
训练
8B、70B 和 405B 基准测试
24.09
堆栈
2304
FP8、BF16
PaXML
训练
50 亿和 1750 亿项基准测试
2004 年 3 月 24 日
合成
2048
FP8、BF16
Maxtext
训练
Llama2 70B 基准测试
2024.12.09
合成
2048
FP8、BF16
Grok1
训练
Grok1 114B 基准测试
24.09
合成
2048
FP8、BF16
Llama 2
微调
Hugging Face 700B 基准测试
24.02
HF Llama2
512
BF16
Mistral
微调
Hugging Face 70B 基准测试
24.02
高频 Mistral
256
BF16
表 1、DGX Cloud Benchmarking Recipes 目录中的可用工作负载
在表 1 中，工作负载包括训练和微调，并尽可能支持 FP8 和 BF16。
每个训练工作负载对于其如何练习平台都有不同的指纹识别。关于工作负载的指纹识别，您可能会问一个基本问题：“计算时间与通信或网络时间的重叠程度如何？”
根据并行性和超参数（如序列长度和批量大小）的选择，一些模型的计算受限程度可能更高，而一些模型的通信受限程度可能更高。扩展行为也会随着 GPU 数量的增加以及扩展类型（弱或强）的增加而在模型之间发生变化。
对于每种工作负载和集群规模，您必须调整模型和系统以获得最佳性能。
在模型方面，这可能涉及调整并行策略、批量大小、精度格式和数据加载策略以及其他配置。在系统端，请确保工作负载充分利用 NVLink 高带宽（例如，用于张量和上下文并行），并确认横向扩展结构不是相应网络集合的阻塞器（例如，用于 pipeline 或 expert 并行）。
后者需要能够提供低传输延迟（RDMA）、有效的拥塞管理和动态路由的网络，如参考
NVIDIA SpectrumX 和 InfiniBand
网络架构中所示。为了高效扩展 AI 工作负载，使用这些技术至关重要，因为它们有助于减轻抖动的影响，确保一致的性能和可靠性。
使用 FP8
DGX 云 Benchmarking Recipes 专为 FP8 工作负载提供优化的配置和调优建议，可帮助您利用这种精度格式实现出色性能。例如，Llama 3.1 70B 训练方法包括针对 DGX 云平台仔细测试和优化的 FP8 设置。
了解给定 AI 工作负载的良好性能的构成要素可能非常复杂。DGX 云基准测试方法为各种热门模型提供一系列基准性能结果，使您能够为自己的实现设定现实的期望和目标。
这些基准包括模型 FLOPS 利用率（MFU）等指标，用于衡量模型使用可用计算资源的效率。您可以看到热门模型的 MFU 和吞吐量比较情况。通过将结果与这些基准进行比较，您可以衡量优化的有效性并找出需要改进的地方。
DeepSeek-R1
是在一个 NVIDIA H200 GPU 节点上运行的 671B 参数模型。它的高计算利用率表明，计算、网络和并行策略的整体优化如何将交付的性能推向理论极限。系统化的基准测试可实现直接比较，帮助团队协作优化模型和平台，从而更大限度地提高 GPU 系统的价值。
最后，这些针对每个工作负载的性能优化还表明，需要围绕应用程序调优开展进一步的研究和讨论。例如，PyTorch 和 pandas 等框架的并行化策略的推荐用法因工作负载和平台的组合而异。
开始使用 DGX Cloud 基准测试方法
平台性能基准测试方法托管在 NVIDIA 的公共注册表 NGC Catalog 中。有关最新版本 recipes 的更多信息，请参阅
DGX Cloud Benchmarking 24.11.1
。
在每个工作负载 recipe 中，您可以访问以下内容：
针对跨环境再现性的容器化基准测试
根据需要生成合成数据的 Scripts
性能指标收集和报告 (发送至
stdout
)
针对每个平台的工作负载的配置最佳实践
用于比较的 NVIDIA 参考架构的性能数据
方法需要 Slurm 集群管理。对 Kubernetes 的支持目前正在开发中。要使用 DGX Cloud Benchmarking Recipes，下载最适合您工作负载的方法，并执行集群设置和基准测试脚本。
继续推进平台性能目标
在当今的 AI 环境中，实现出色性能需要超越单个组件的视野，以了解整个系统如何协同工作。虽然原始 GPU 功能很重要，但完整的优化来自仔细调整堆栈的每一层，从硬件和软件配置到特定工作负载的参数。
在 NVIDIA，我们使用基准测试方法来不断优化技术堆栈的每一层，从
NVIDIA NVLink 和 NVLink Switch
等硬件互连到
NVIDIA TensorRT-LLM
等软件库，从而随着时间的推移实现显著的性能提升。
例如，仅通过持续改进软件开发，
NVIDIA H100 GPU 上的 MLPerf 推理在短短一年内就将加速计算性能提高了 3.4 倍
。这些持续优化使组织能够运行更复杂的模型、降低基础架构需求、提高效率，从而推动进一步创新。
这些基准测试方法使您的团队能够：
针对特定环境 (包括 FP8) 优化 AI 工作负载。
评估集群性能与 NVIDIA 观察到的性能的接近程度。
识别当前设置中的性能瓶颈。
训练大型模型可能需要数周或数月时间，并且会耗费数百万计算资源，因此适度的性能改进可以转化为大幅节省时间和成本。通过使用 NVIDIA 不断改进的性能优化和特定于工作负载的方法，您的组织可以更大限度地投资 AI 基础架构，并将工程工作重点放在创新上，而不是基础架构调优。
有关更多信息，请参阅
DGX Cloud 基准测试方法
。