Advancing Physical AI with NVIDIA Cosmos World Foundation Model Platform
As robotics and autonomous vehicles advance, accelerating development of
physical AI
—which enables
autonomous machines
to perceive, understand, and perform complex actions in the physical world—has become essential. At the center of these systems are
world foundation models (WFMs)
—AI models that simulate physical states through physics-aware videos, enabling machines to make accurate decisions and interact seamlessly with their surroundings.
NVIDIA Cosmos
is a platform that helps developers build custom world models for physical AI systems at scale. It offers
open world foundation models
and tools for every stage of development, from data curation to training to customization.
Video 1. NVIDIA Cosmos: A World Foundation Model Platform for Physical AI
This post explains Cosmos and its key features that accelerate physical AI development.
Accelerating world model development with NVIDIA Cosmos
Building physical AI is challenging, demanding precise simulations and real-world behavior understanding and prediction. A key tool for overcoming these challenges is a world model, which predicts future environmental states based on past observations and current inputs. These models are invaluable for physical AI builders, enabling them to simulate, train, and refine systems in controlled environments.
However, developing effective world models requires vast amounts of data, computational power, and real-world testing, which can introduce significant safety risks, logistical hurdles, and prohibitive costs. To address these challenges, developers often turn to
synthetic data
generated from
3D simulations
to train models. While synthetic data is a powerful tool, creating it is resource-intensive and may fall short of accurately reflecting real-world physics, particularly in complex or edge-case scenarios.
The
end-to-end NVIDIA Cosmos platform
accelerates world model development for physical AI systems. Built on CUDA, Cosmos combines state-of-the-art world foundation models, video tokenizers, and AI-accelerated data processing pipelines.
Developers can accelerate world model development by fine-tuning Cosmos world foundation models or building new ones from the ground up. In addition to Cosmos world foundation models, the platform also includes:
NVIDIA NeMo Curator
for efficient video data curation
Cosmos Tokenizer
for efficient, compact, and high-fidelity video tokenization
Cosmos world foundation models pretrained for robotics and autonomous driving applications
NVIDIA NeMo Framework
for model training and optimization
Figure 1. NVIDIA Cosmos is a world foundation model development platform with generative models, data curator, tokenizer, and framework to accelerate physical AI development
Pretrained world foundation models for physical AI
Cosmos world foundation models are pretrained large generative AI models trained on 9,000 trillion tokens—including 20 million hours of data from
autonomous driving
,
robotics
, synthetic environments, and other related domains. These models create realistic synthetic videos of environments and interactions, providing a scalable foundation for training complex systems, from simulating
humanoid robots
performing advanced actions to developing end-to-end autonomous driving models.
These models use two architectures: autoregressive and diffusion. Both approaches use the transformer architecture for its scalability and effectiveness in handling complex temporal dependencies.
Autoregressive model
Cosmos autoregressive model is designed for video generation, predicting the next token based on input text and past video frames. It uses a transformer decoder architecture, with key modifications for world model development.
3D RoPE (Rotary Position Embeddings) encodes spatial and temporal dimensions separately, ensuring precise video sequence representation.
Cross-attention layers enable text inputs, providing better control over world generation.
QK-normalization enhances training stability.
Pretraining of this model is progressive, starting with predicting up to 17 future frames from a single input frame, then extending to 34 frames, and eventually up to 121 frames (or 50,000 tokens). Text inputs are introduced to combine descriptions with video frames, and the model is fine-tuned with high-quality data for robust performance. This structured approach enables the model to generate videos of varying lengths and complexities, with or without text inputs.
Figure 2. The Cosmos autoregressive model uses a transformer decoder architecture, with key modifications for world model development
Diffusion models
Diffusion models
are popular for generating images, videos, and audio due to their ability to deconstruct training data and reconstruct it based on user input, producing high-quality, realistic outputs.
Diffusion models operate in two phases:
Forward diffusion process:
Training data is progressively corrupted by adding Gaussian noise over multiple steps, effectively transforming it into pure noise.
Reverse diffusion process:
The model learns to reverse this noise step by step, recovering the original data by denoising the corrupted input.
Once trained, diffusion models generate new data by sampling random Gaussian noise and passing it through the learned denoising process. In addition, Cosmos diffusion models also get several key updates tailored for physical AI development.
3D patchification processes video into smaller patches, simplifying spatio-temporal sequence representation.
Hybrid positional embeddings handle spatial and temporal dimensions, supporting videos with varying resolutions and frame rates.
Cross-attention layers incorporate text inputs, enabling better control over video generation based on descriptions.
Adaptive layer normalization with LoRA reduces model size by 36%, maintaining high performance with fewer resources.
Figure 3. Cosmos diffusion model architecture combines advanced video compression, flexible positional encoding, and text integration to deliver high-quality, physics-aware video generation
Model sizes for varied needs
Developers can choose from the following three model sizes to meet performance, quality, and deployment needs.
Nano:
Optimized for real-time, low-latency inference and edge deployment.
Super:
Designed as performant baseline models.
Ultra:
Focused on maximum quality and fidelity, ideal for distilling custom models.
Strengths and limitations
Cosmos world foundation models generate low-resolution, real-world-accurate synthetic video data, essential for training
robotics
and autonomous vehicle systems. While they lack artistic flair, their outputs closely replicate the physical world, making them ideal for precise object permanence and realistic scenarios in physical AI model training.
Guardrails for safe use of Cosmos world foundation models
AI models need guardrails to ensure reliability by mitigating hallucinations, preventing harmful outputs, safeguarding privacy, and aligning with AI standards for safe and controlled deployment. Cosmos ensures the safe use of its world foundation models through a customizable, two-stage guardrail system aligned with NVIDIA’s commitment to trustworthy AI.
Cosmos Guardrails operates in two stages: Pre-guard and Post-guard.
Pre-guard
This stage involves text prompt-based safety measures using two layers:
Keyword Blocking:
A blocklist checker scans prompts for unsafe keywords, using lemmatization to detect variations and blocking non-English terms or spelling errors.
Aegis Guardrail:
The NVIDIA fine-tuned Aegis AI Content Safety model detects and blocks semantically unsafe prompts, including categories like violence, harassment, and profanity. Unsafe prompts halt video generation and return an error message.
Post-guard
The Post-guard stage ensures the safety of generated videos through:
Video Content Safety Classifier:
A multiclass classifier evaluates every video frame for safety. If any frame is flagged as unsafe, the entire video is rejected.
Face Blur Filter:
All human faces in generated videos are blurred using the RetinaFace model to protect privacy and reduce biases based on age, gender, or race.
NVIDIA experts rigorously test with adversarial examples, annotating over 10,000 prompt-video pairs to refine the system and address edge cases.
Evaluating Cosmos world foundation models for 3D consistency and physics alignment
Cosmos benchmarks play a crucial role in assessing the ability of world foundation models to simulate real-world physics accurately and efficiently for physical AI applications. While publicly available benchmarks for video generation focus on fidelity, temporal consistency, and speed of generated videos, Cosmos benchmarks add new dimensions to evaluate generalist models: 3D consistency and physics alignment, ensuring the videos are evaluated based on accuracy required for physical AI systems.
3D consistency
Cosmos models were tested for 3D consistency on static scenes from a curated subset of 500 videos from an
open dataset
. Text prompts describing the videos were generated to avoid motion-related complexities. Comparisons were made against
VideoLDM
, a baseline generative model.
Metrics used
Geometric Consistency:
Assessed through epipolar geometry constraints using metrics like Sampson error and camera pose estimation success rate.
View Synthesis Consistency:
Evaluated through metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). These metrics measure the quality of synthesized views from interpolated camera positions.
Lower Sampson error and higher success rates indicate better 3D alignment. Similarly, higher PSNR and SSIM and lower LPIPS are indicators of a better quality.
Model
Sampson Error ↓
Pose Estimation Success Rate (%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
VideoLDM
0.841
4.40%
26.23
0.783
0.135
Cosmos 1.0 Diffusion Text2World 7B
0.355
62.60%
33.02
0.939
0.070
Cosmos 1.0 Diffusion Video2World 7B
0.473
68.40%
30.66
0.929
0.085
Cosmos 1.0 Autoregressive 4B
0.433
35.60%
32.56
0.933
0.090
Cosmos 1.0 Autoregressive Video2World 5B
0.392
27.00%
32.18
0.931
0.090
Real videos (reference)
0.431
56.40%
35.38
0.962
0.054
Table 1. Evaluation of 3D consistency for Cosmos world foundation models versus base VideoLDM model
Results
Cosmos world foundation models outperform the baseline in 3D consistency (table 1), with higher geometric alignment and camera pose success rates. Their synthesized views match real-world quality, confirming their effectiveness as world simulators.
Physical alignment
Physics alignment tests how well Cosmos models simulate real-world physics, including motion, gravity, and energy dynamics. Using
NVIDIA PhysX
and
NVIDIA Isaac Sim
, eight controlled scenarios were designed to evaluate properties like gravity, collision, torque, and inertia in virtual environments.
Metrics used
Pixel-Level Metrics:
Peak Signal-to-Noise Ratio (PSNR) measures how closely the pixel values of the model’s output match the reference video. Higher values indicate less noise and better accuracy. Structural Similarity Index Measure (SSIM) assesses the similarity in structure, luminance, and contrast between the generated and ground-truth frames. Higher SSIM values reflect greater visual fidelity.
Feature-Level Metric:
DreamSim measures the similarity between high-level features extracted from both videos. This approach evaluates the semantic consistency of the generated content, focusing on objects and motion rather than individual pixels.
Object-Level Metric:
Intersection-over-Union (IoU) calculates the overlap between the predicted and actual object regions in the video. This is especially useful for tracking specific objects through the simulation to ensure their behavior aligns with physical expectations.
Higher PSNR, SSIM, DreamSim and IoU are indicators of better physical alignment.
Model
Conditioning
PSNR ↑
SSIM ↑
DreamSim ↑
Avg. IoU ↑
Cosmos 1.0 Diffusion Video2World 7B
prompt + 1 frame
17.34
0.54
0.84
0.332
Cosmos 1.0 Diffusion Video2World 7B
prompt + 9 frames
21.06
0.69
0.86
0.592
Cosmos 1.0 Diffusion Video2World 14B
prompt + 1 frame
16.81
0.52
0.84
0.338
Cosmos 1.0 Diffusion Video2World 14B
prompt + 9 frames
20.21
0.64
0.86
0.598
Cosmos 1.0 Autoregressive 4B
1 frame
17.91
0.49
0.83
0.394
Cosmos 1.0 Autoregressive 4B
9 frames
18.13
0.48
0.86
0.481
Cosmos 1.0 Autoregressive Video2World 5B
prompt + 1 frame
17.67
0.48
0.82
0.376
Cosmos 1.0 Autoregressive Video2World 5B
prompt + 9 frames
18.29
0.48
0.86
0.481
Cosmos 1.0 Autoregressive Video2World 12B
1 frame
17.94
0.49
0.83
0.395
Cosmos 1.0 Autoregressive Video2World 12B
9 frames
18.22
0.49
0.87
0.487
Cosmos 1.0 Autoregressive Video2World 13B
prompt + 1 frame
18
0.49
0.83
0.397
Cosmos 1.0 Autoregressive Video2World 13B
prompt + 9 frames
18.26
0.48
0.87
0.482
Table 2. Physics alignment results with metrics calculated over 33 frames, the maximum length supported by the autoregressive variants of the Cosmos world foundation models
Results
Cosmos world foundation models show strong adherence to physical laws (Table 2), particularly with increased conditioning data. Post-training on camera conditioning dataset achieves a twofold increase in pose estimation success rate compared to baseline models. However, challenges like object impermanence (where objects vanish or appear unexpectedly) and implausible behaviors (such as violating gravity) highlight areas for improvement.
Customizing for physical AI applications with Cosmos and NVIDIA Omniverse
Video search and understanding:
Simplifies video tagging and search by understanding spatial and temporal patterns, making training data preparation easier.
Controllable 3D-to-real synthetic data generation:
With
NVIDIA Omniverse
, developers can create 3D scenarios and use Cosmos to generate photorealistic videos that are precisely controlled by 3D scenes for highly tailored synthetic datasets.
Policy model development and evaluation:
World foundation models fine-tuned for action-conditioned video prediction enable scalable, reproducible evaluation of policy models—strategies mapping states to actions—reducing reliance on risky real-world tests or complex simulations for tasks like obstacle navigation or object manipulation.
Foresight for action selection:
Cosmos equips physical AI models with predictive capabilities to assess the outcomes of potential actions.
Multiverse simulation:
Using Cosmos and NVIDIA Omniverse, developers can simulate multiple future outcomes to help AI models evaluate and select the best strategy for achieving its goals, benefiting applications like predictive maintenance and autonomous decision-making.
From generalist to customized specialist models
Cosmos introduces a two-stage approach to world model training.
Generalist models:
Cosmos world foundation models are built as generalists, trained on extensive datasets that encompass diverse real-world physics and environments. These open models are capable of handling a broad range of scenarios, from natural dynamics to robotic interactions, providing a solid foundation for any physical AI task.
Specialist models:
Developers can fine-tune generalist models using smaller, targeted datasets to create specialists tailored for specific applications, such as autonomous driving or humanoid robotics or they can generate customized synthetic scenarios, such as night scenes with emergency vehicles or high-fidelity industrial robotics environments. This fine-tuning process significantly reduces the required data and training time compared to training models from scratch.
Cosmos accelerates training and fine-tuning with efficient video processing pipelines, highly performant tokenizer, and advanced training frameworks, enabling developers to address operational needs and edge cases for advancing physical AI.
Accelerated data processing with NVIDIA NeMo Curator
Training models require curated, high-quality data, which is time and resource-intensive. NVIDIA Cosmos includes a data processing and curation pipeline powered by NVIDIA NeMo Curator and optimized for NVIDIA data center GPUs.
NVIDIA NeMo Curator enables robotics and AV developers to process vast datasets efficiently. For example, 20 million hours of video can be processed in 40 days on NVIDIA Hopper GPUs, or just 14 days on NVIDIA Blackwell GPUs—compared to 3.4 years on unoptimized CPU pipelines.
Key benefits include:
89x faster curation:
Dramatically reduces processing time
Scalability:
Handles 100+ PB of data seamlessly
High throughput:
Advanced filtering, captioning, and embedding ensure quality without sacrificing speed
Figure 4. Cosmos includes NeMo Curator that delivers 89x faster video data processing
High-fidelity compression and reconstruction with Cosmos Tokenizer
After data is curated, it must be tokenized for training. Tokenization breaks down complex data into manageable units, enabling models to process and learn from it more efficiently.
Cosmos tokenizers simplify this process with faster compression and visual reconstruction while preserving quality, reducing costs and complexity. For autoregressive models, the discrete tokenizer compresses data 8x in time and 16×16 in space, processing up to 49 frames at once. For diffusion models, the continuous tokenizer achieves 8x time and 8×8 space compression, handling up to 121 frames.
Fine-tuning with NVIDIA NeMo
Developers can fine-tune Cosmos world foundation models using the
NVIDIA NeMo
Framework. NeMo Framework accelerates model training on GPU-powered systems, whether enhancing an existing model or building a new one, from on-premises data centers to the cloud.
NeMo Framework efficiently loads multimodal data by:
Sharding terabyte size dataset into compressed files to reduce IO overhead.
Deterministically saving and loading datasets to avoid repetition and minimize compute waste.
Reducing network bandwidth when exchanging data using optimized communications.
Get started with NVIDIA Cosmos
Cosmos world foundation models are open and available on
NGC
and
Hugging Face
. Developers can also run Cosmos world foundation models on the
NVIDIA API catalog
. Also available on the
API catalog
are Cosmos tools to enhance text prompts for accuracy, an inbuilt watermarking system that enables easy future identification of AI-generated sequences, and a specialized model to decode video sequences for augmented reality applications. To learn more,
watch the demo
.
NeMo Curator
for accelerated data processing pipelines is available as a managed service and SDK. Developers can now
apply for early access
. Cosmos tokenizers are open neural networks available on
GitHub
and
Hugging Face
.
Get started with NVIDIA Cosmos
.
NVIDIA Cosmos World 基础模型平台助力物理 AI 进步
随着机器人和
自动驾驶汽车
的发展，加速
物理 AI
的发展变得至关重要，而物理 AI 使自主机器能够感知、理解并在现实世界中执行复杂的操作。这些系统的核心是
世界基础模型 (WFMs)
，即通过物理感知视频模拟物理状态的 AI 模型，使机器能够做出准确决策并与周围环境无缝交互。
NVIDIA Cosmos
平台可帮助开发者大规模为物理 AI 系统构建自定义世界模型。它为从数据管护、训练到定制的每个开发阶段提供开放世界基础模型和工具。
视频 1、NVIDIA Cosmos：适用于物理 AI 的世界基础模型平台
本文将介绍 Cosmos 及其加速物理 AI 开发的主要功能。Cosmos 是一种开源的 Python 库，用于加速物理 AI 开发。
借助 NVIDIA Cosmos 加速世界模型开发
构建物理 AI 极具挑战性，需要精确的模拟以及真实世界的行为理解和预测。克服这些挑战的一个关键工具是世界模型，该模型根据过去的观察结果和当前的输入结果预测未来的环境状态。这些模型对于物理 AI 构建者来说非常重要，使他们能够在受控环境中模拟、训练和优化系统。
然而，开发有效的世界模型需要大量数据、计算能力和真实世界的测试，这可能会带来重大的安全风险、物流障碍和高昂的成本。为了应对这些挑战，开发者通常会使用通过 3D 仿真生成的
合成数据
来训练模型。虽然合成数据是一种功能强大的工具，但创建合成数据需要大量资源，可能无法准确反映现实世界的物理特性，在复杂或边缘情况下尤其如此。
端到端
NVIDIA Cosmos 平台
可加速物理 AI 系统的世界模型开发。Cosmos 基于 CUDA 构建，结合了先进的世界基础模型、视频标记器和 AI 加速的数据处理流程。
开发者可以通过微调 Cosmos 世界基础模型或从头开始构建新模型来加速世界模型的开发。除了 Cosmos 世界基础模型之外，该平台还包括：
用于高效视频数据管护的
NVIDIA NeMo Curator
Cosmos Tokenizer
可实现高效、紧凑和高保真的视频标记化
为机器人和自动驾驶应用预训练的 Cosmos World Foundation 模型
用于模型训练和优化的
NVIDIA NeMo 框架
图 1、NVIDIA Cosmos 是一个世界性的基础模型开发平台，包含生成式模型、数据管护器、标记器和框架，可加速物理 AI 开发
适用于物理 AI 的
预训练世界基础模型
Cosmos 世界基础模型是预训练的大型生成式 AI 模型，使用 9000 万亿个令牌进行训练，其中包括来自
自动驾驶
、
机器人
开发、合成环境和其他相关领域的 2000 万个小时的数据。这些模型可创建有关环境和交互的逼真合成视频，为训练复杂系统 (从模拟执行高级动作的类
人型机器人
到开发端到端自动驾驶模型) 提供可扩展的基础。
这些模型使用两种架构：自回归和扩散。这两种方法都使用 Transformer 架构，因为其可扩展性和有效性可用于处理复杂的时间依赖项。
自回归模型
Cosmos 自回归模型专为视频生成而设计，可根据输入文本和过去的视频帧预测下一个令牌。它使用 Transformer 解码器架构，并针对世界模型开发进行了关键修改。
3D RoPE (Rotary Position Embeddings) 可分别对空间和时间维度进行编码，确保精确的视频序列表示。
交叉注意力层支持文本输入，从而更好地控制世界生成。
QK 标准化可增强训练稳定性。
此模型的预训练是渐进式的，首先从单个输入帧预测多达 17 帧的未来帧，然后扩展到 34 帧，最终达到 121 帧 (或 50,000 个 tokens)。引入文本输入以将描述与视频帧相结合，并使用高质量数据对模型进行微调，以实现强大的性能。这种结构化方法使模型能够生成长度和复杂性各不相同的视频，无论是否输入文本。
图 2、Cosmos 自回归模型使用 Transformer 解码器架构，并对世界模型开发进行了关键修改
扩散模型
Diffusion 模型在生成图像、视频和音频方面很受欢迎，因为它们能够解构训练数据并根据用户输入进行重建，从而生成高质量、逼真的输出。
扩散模型分为两个阶段：
前向扩散过程：
通过在多个步骤中添加高斯噪声，有效地将其转换为纯噪声，训练数据会逐渐损坏。
反向扩散过程：
模型会逐步学习反向噪声，通过对损坏的输入降噪来恢复原始数据，例如使用 Stable Diffusion 模型。
经过训练后，扩散模型通过对随机高斯噪声进行采样并将其传递给学习的降噪过程来生成新数据。此外，Cosmos 扩散模型还获得了一些专为物理 AI 开发打造的关键更新。
3D Patchification 将视频处理为更小的分块，从而简化时空序列表示。
混合位置嵌入可处理空间和时间维度，支持具有不同分辨率和帧率的视频。
交叉注意力层整合文本输入，从而更好地控制基于描述生成的视频。
采用 LoRA 的自适应层归一化可将模型大小减少 36%，从而以更少的资源保持高性能。
图 3、Cosmos 扩散模型架构将先进的视频压缩、灵活的位置编码和文本集成相结合，可生成具有物理感知的高质量视频
满足不同需求的模型大小
开发者可以从以下三种模型大小中进行选择，以满足性能、质量和部署需求。
Nano：
针对实时、低延迟推理和边缘部署进行优化。
Super：
设计为高性能基准模型。
Ultra：专注于最大化质量和保真度，适合提取自定义模型。
优点和局限性
Cosmos 世界基础模型生成低分辨率、真实世界准确的合成视频数据，这对于训练
机器人
和自动驾驶汽车系统至关重要。虽然它们缺乏艺术天赋，但其输出结果密切复制了物理世界，因此非常适合在物理 AI 模型训练中实现精确的物体持久性和逼真的场景。
用于安全使用 Cosmos World 基础模型的护栏
AI 模型需要护栏来确保可靠性，具体方法包括减少幻影、防止有害输出、保护隐私，以及符合 AI 标准以实现安全可控的部署。Cosmos 通过可定制的双阶段护栏系统确保其 World Foundation 模型的安全使用，该系统符合 NVIDIA 对值得信赖的 AI 的承诺。
Cosmos Guardrails 分为两个阶段：Pre-guard 和 Post-guard。
预防护
此阶段涉及基于文本提示的安全措施，使用两层：
关键字屏蔽：屏蔽列表检查器会扫描不安全关键字的提示，使用引言化来检测变异，并阻止非英语术语或拼写错误。
Aegis Guardrail：
经过 NVIDIA 微调的 Aegis AI 内容安全模型可以检测并阻止语义上不安全的提示，包括暴力、骚扰和冒犯等类别。不安全提示停止视频生成并返回错误消息。
护卫队队员
Post-guard Stage 通过以下方式确保生成视频的安全性：
视频内容安全分类器：
多类别分类器评估每个视频帧的安全性。如果任何帧被标记为不安全，则整个视频将被拒绝。
面部模糊滤镜：
使用 RetinaFace 模型对生成的视频中的所有人脸进行模糊处理，以保护隐私并减少基于年龄、性别或种族的偏见。
NVIDIA 专家使用对抗示例进行严格测试，标注超过 10,000 个提示视频对，以优化系统并解决边缘案例。
评估 Cosmos 世界基础模型，以实现 3D 一致性和物理对齐
Cosmos 基准测试在评估世界基础模型为物理 AI 应用准确高效地模拟真实物理的能力方面发挥着至关重要的作用。虽然公开可用的视频生成基准测试侧重于保真度、时间一致性和生成视频的速度，但 Cosmos 基准测试增加了新的维度来评估通用模型：3D 一致性和物理对齐，确保根据物理 AI 系统所需的准确性评估视频。
3D 一致性
Cosmos 模型在
开放数据集
中的 500 个精选视频子集中进行了静态场景的 3D 一致性测试。生成描述视频的文本提示是为了避免与动作相关的复杂性。并与基准生成模型
VideoLDM
进行了比较。
所用指标
几何一致性：使用 Sampson 误差和摄像头姿态估计成功率等指标通过外极性几何约束条件进行评估。
查看合成一致性：
通过峰值信噪比 (Peak Signal-to-Noise Ratio, PSNR)、结构相似性指数 (Structural Similarity Index, SSIM) 和学习感知图像块相似性 (Learned Perceptual Image Patch Similarity, LPIPS) 等指标进行评估。这些指标用于测量从插入的摄像头位置合成视图的质量。
更低的 Sampson 误差和更高的成功率表明 3D 对齐效果更好。同样，较高的 PSNR 和 SSIM 以及较低的 LPIPS 都表示质量有所提高。
模型
Sampson 错误*
姿态估计成功率 (%)*
PSNR*
SSIM
LPIPS
VideoLDM
0.841
4.40%
26.23
0.783
0.135
Cosmos 1.0 Diffusion Text2World 7B
0.355
62.60%
33.02
0.939
0.070
Cosmos 1.0 Diffusion Video2World 7B
0.473
68.40%
30.66
0.929
0.085
Cosmos 1.0 Autoregressive 4B
0.433
35.60%
32.56
0.933
0.090
Cosmos 1.0 Autoregressive Video2World 5B
0.392
27.00%
32.18
0.931
0.090
真实视频 (参考)
0.431
56.40%
35.38
0.962
0.054
表 1、评估 Cosmos 世界基础模型与基础 VideoLDM 模型的 3D 一致性
成果
Cosmos 世界基础模型在 3D 一致性方面的表现优于基准 (表 1)，具有更高的几何对齐和摄像头姿态成功率。其合成视图与真实世界的质量相匹配，证实了其作为世界模拟器的有效性。
物理对齐
物理对齐测试 Cosmos 模型在模拟真实物理 (包括运动、重力和能量动力学) 时的效果。借助
NVIDIA PhysX
和
NVIDIA Isaac Sim
，设计出八个受控场景，用于评估虚拟环境中的重力、碰撞、扭矩和惯性等属性。
所用指标
像素级指标
：峰值信噪比（Peak Signal-to-Noise Ratio，PSNR）可测量模型输出的像素值与参考视频的匹配程度。值越高表示噪点越小，准确性越高。结构相似性指数指标（Structural Similarity Index Measure，SSIM）用于评估生成的帧与真实帧之间在结构、亮度和对比度方面的相似性。较高的 SSIM 值反映了更高的视觉保真度。
特征级别指标
：DreamSim 测量从两个视频中提取的高级特征之间的相似性。这种方法评估生成内容的语义一致性，重点关注物体和运动，而不是单个像素。
目标级别指标：
交集并集比 (Intersection-over-Union，IoU) 用于计算视频中预测的目标区域与实际目标区域之间的重叠。这对于在整个模拟过程中追踪特定物体特别有用，可确保其行为符合物理预期。
PSNR、SSIM、DreamSim 和 IoU 越高，表示物理对齐越好。
模型
调节
PSNR*
SSIM
DreamSim™
平均值。IoU*
Cosmos 1.0 Diffusion Video2World 7B
提示符 = 1 帧
17.34
0.54
0.84
0.332
Cosmos 1.0 Diffusion Video2World 7B
提示符 = 9 帧
21.06
0.69
0.86
0.592
Cosmos 1.0 Diffusion Video2World 14B
提示符 = 1 帧
16.81
0.52
0.84
0.338
Cosmos 1.0 Diffusion Video2World 14B
提示符 = 9 帧
20.21
0.64
0.86
0.598
Cosmos 1.0 Autoregressive 4B
1 帧
17.91
0.49
0.83
0.394
Cosmos 1.0 Autoregressive 4B
9 帧
18.13
0.48
0.86
0.481
Cosmos 1.0 Autoregressive Video2World 5B
提示符 = 1 帧
17.67
0.48
0.82
0.376
Cosmos 1.0 Autoregressive Video2World 5B
提示符 = 9 帧
18.29
0.48
0.86
0.481
Cosmos 1.0 Autoregressive Video2World 12B
1 帧
17.94
0.49
0.83
0.395
Cosmos 1.0 Autoregressive Video2World 12B
9 帧
18.22
0.49
0.87
0.487
Cosmos 1.0 Autoregressive Video2World 13B
提示符 = 1 帧
18
0.49
0.83
0.397
Cosmos 1.0 Autoregressive Video2World 13B
提示符 = 9 帧
18.26
0.48
0.87
0.482
表 2、物理对齐结果，计算指标超过 33 帧 (Cosmos 世界基础模型的自回归变体支持的最大长度)
成果
Cosmos 世界基础模型非常遵守物理定律 (表 2)，尤其是在条件数据增加的情况下。与基线模型相比，在摄像头调节数据集上进行后训练后，姿态估计的成功率提高了两倍。然而，物体无常 (物体意外消失或出现) 和难以置信的行为 (如违反重力) 等挑战凸显了需要改进的领域。
使用 Cosmos 和 NVIDIA Omniverse 定制物理 AI 应用
视频搜索和理解：
通过了解空间和时间模式来简化视频标记和搜索，从而更轻松地准备训练数据
可控的 3D 到真实合成数据生成：
借助
NVIDIA Omniverse
，开发者可以创建 3D 场景，并使用 Cosmos 生成逼真的视频，这些视频由 3D 场景精确控制，用于高度定制的合成数据集。
策略模型开发和评估：
World foundation models 针对动作条件视频预测进行了微调，可实现策略模型的可扩展、可再现评估 (将状态映射到动作的策略)，从而减少对障碍物导航或对象操作等任务进行风险真实测试或复杂模拟的依赖。
行动选择前瞻：
Cosmos 为物理 AI 模型配备预测功能，以评估潜在行动的结果。
Multiverse 模拟：
借助 Cosmos 和 NVIDIA Omniverse，开发者可以模拟多种未来结果，帮助 AI 模型评估和选择实现目标的最佳策略，从而使预测性维护和自主决策等应用受益。
从通用模型到定制的专业模型
Cosmos 为世界模型训练引入了一种分为两个阶段的方法。
多面手模型：Cosmos 世界基础模型以多面手的身份构建，基于涵盖各种真实物理和环境的大量数据集进行训练。这些开放模型能够处理从自然动力学到机器人交互的各种场景，为任何物理 AI 任务提供坚实的基础。
专业模型：开发者可以使用更小的定向数据集微调通用模型，为特定应用（如自动驾驶或人形机器人）打造量身定制的专家模型，也可以生成定制的合成场景，如夜间场景中的紧急车辆或高保真工业机器人环境。与从头开始训练模型相比，这种微调过程可显著减少所需的数据和训练时间。
Cosmos 通过高效的视频处理流程、高性能的 tokenizer 和高级训练框架加速训练和微调，使开发者能够满足运营需求和边缘案例，推动物理 AI 的发展。
借助 NVIDIA NeMo Curator 加速数据处理
训练模型需要精心挑选的高质量数据，而且需要耗费大量时间和资源。NVIDIA Cosmos 包含由 NVIDIA NeMo Curator 提供支持并针对 NVIDIA 数据中心 GPU 进行优化的数据处理和管护流程。
借助 NVIDIA NeMo Curator，机器人和自动驾驶（AV）开发者能够高效处理庞大的数据集。例如，20 million 小时的视频可以在 40 天内在 NVIDIA Hopper GPU 上处理完毕，或仅在 14 天内在 NVIDIA Blackwell GPU 上处理完毕——相比之下，未经优化的 CPU 流水线需要 3.4 年。
主要优势包括：
PyTorch、pandas、LangChain、Megatron、NVIDIA、cuOpt、Stable Diffusion、Llama、Jetson、Google、Python、Hugging Face、Arm、Anaconda、Siemens、DPU、GPU 和 Github 等技术的
89
倍管护速度：显著缩短处理时间
可扩展性：
无缝处理 100 多个 PB 的数据
高吞吐量：高级过滤、字幕和嵌入可在不牺牲速度的情况下确保质量
图 4、Cosmos 包含 NeMo Curator，可将视频数据处理速度提高 89 倍
使用 Cosmos Tokenizer 实现高保真压缩和重建
整理数据后，必须对其进行标记化以进行训练。标记化将复杂的数据分解为可管理的单元，使模型能够更高效地处理数据并从中学习。
Cosmos 标记器可简化此过程，加快压缩和视觉重建速度，同时保持质量，降低成本和复杂性。对于自回归模型，离散分词器可将数据压缩为时间的 8 倍，在空间中压缩为 16 × 16，一次最多处理 49 帧。对于扩散模型，连续分词器可实现 8 倍的时间和 8 × 8 的空间压缩，最多可处理 121 帧。
使用 NVIDIA NeMo 进行微调
开发者可以使用
NVIDIA NeMo
框架微调 Cosmos 世界基础模型。NeMo 框架可以在 GPU 驱动的系统上加速模型训练，无论是增强现有模型还是构建新模型，从本地数据中心到云端。
NeMo 框架通过以下方式高效加载多模态数据：
将 TB 大小的数据集分片到压缩文件中，以减少 IO 开销。
决定性地保存和加载数据集，以避免重复并尽可能减少计算浪费。
在使用优化的通信交换数据时降低网络带宽。
开始使用 NVIDIA Cosmos
Cosmos World Foundation 模型现已开放，可在
NGC
和
Hugging Face
上获取。开发者还可以在
NVIDIA
API Catalog
上运行 Cosmos World Foundation 模型。API Catalog 上还提供了用于增强文本提示以提高准确性的 Cosmos 工具、支持未来轻松识别 AI 生成序列的内置水印系统，以及用于解码视频序列以用于增强现实应用的专用模型。如需了解详情，请观看演示。
NeMo Curator
for accelerated data processing pipelines is available as a managed service and SDK. Developers can now
apply for early access
. Cosmos tokenizers are open neural networks available on
GitHub
and
Hugging Face
.
开始使用 NVIDIA Cosmos
。