Editor’s note: This post was updated on Friday, Jan. 10, with Best of CES Awards results.
NVIDIA Cosmos
, a platform for accelerating
physical AI
development, introduces a family of
world foundation models
— neural networks that can predict and generate physics-aware videos of the future state of a virtual environment — to help developers build next-generation robots and autonomous vehicles (AVs).
World foundation models, or WFMs, are as fundamental as large language models. They use input data, including text, image, video and movement, to generate and simulate virtual worlds in a way that accurately models the spatial relationships of objects in the scene and their physical interactions.
Announced at CES
, NVIDIA is making available the first wave of Cosmos WFMs for physics-based simulation and synthetic data generation — plus state-of-the-art tokenizers, guardrails, an accelerated data processing and curation pipeline, and a framework for model customization and optimization.
Cosmos won Best AI and Best Overall accolades from the
Best of CES Awards
by the CNET Group, awards partner for the Consumer Technology Association, which produces CES.
Researchers and developers, regardless of their company size, can freely use the Cosmos models under NVIDIA’s permissive open model license that allows commercial usage. Enterprises building AI agents can also use new open
NVIDIA Llama Nemotron and Cosmos Nemotron models
, unveiled at CES.
The openness of Cosmos’ state-of-the-art models unblocks
physical AI
developers building robotics and AV technology and enables enterprises of all sizes to more quickly bring their physical AI applications to market. Developers can use Cosmos models directly to generate physics-based synthetic data, or they can harness the
NVIDIA NeMo framework
to fine-tune the models with their own videos for specific physical AI setups.
Physical AI leaders — including robotics companies 1X, Agility Robotics and XPENG, and AV developers Uber and Waabi  — are already working with Cosmos to accelerate and enhance model development.
Developers can preview the first Cosmos
autoregressive
and
diffusion
models on the
NVIDIA API catalog
, and download the family of models and fine-tuning framework from the
NVIDIA NGC catalog
and
Hugging Face
.
World Foundational Models for Physical AI
Cosmos world foundation models are a suite of open diffusion and autoregressive transformer models for physics-aware video generation. The models have been trained on 9,000 trillion tokens from 20 million hours of real-world human interactions, environment, industrial, robotics and driving data.
The models come in three categories: Nano, for models optimized for real-time,
low-latency inference
and edge deployment; Super, for highly performant baseline models; and Ultra, for maximum quality and fidelity, best used for distilling custom models.
When paired with
NVIDIA Omniverse
3D outputs, the diffusion models generate controllable, high-quality synthetic video data to bootstrap training of robotic and AV perception models. The autoregressive models predict what should come next in a sequence of video frames based on input frames and text. This enables real-time next-token prediction, giving physical AI models the foresight to predict their next best action.
Developers can use Cosmos’ open models for text-to-world and video-to-world generation. Versions of the diffusion and autoregressive models, with between 4 and 14 billion parameters each, are available now on the NGC catalog and
Hugging Face
.
Also available are a 12-billion-parameter upsampling model for refining text prompts, a 7-billion-parameter video decoder optimized for augmented reality, and guardrail models to ensure responsible, safe use.
To demonstrate opportunities for customization, NVIDIA is also releasing fine-tuned model samples for vertical applications, such as generating multisensor views for AVs.
Advancing Robotics, Autonomous Vehicle Applications
Cosmos world foundation models can enable
synthetic data generation
to augment training datasets, simulation to test and debug physical AI models before they’re deployed in the real world, and reinforcement learning in virtual environments to accelerate
AI agent learning
.
Developers can generate massive amounts of controllable, physics-based synthetic data by conditioning Cosmos with composed 3D scenes from NVIDIA Omniverse.
Waabi, a company pioneering generative AI for the physical world, starting with autonomous vehicles, is evaluating the use of Cosmos for the search and curation of data for AV software development and simulation. This will further accelerate the company’s industry-leading approach to safety, which is based on Waabi World, a generative AI simulator that can create any situation a vehicle might encounter with the same level of realism as if it happened in the real world.
In robotics, WFMs can generate synthetic virtual environments or worlds to provide a less expensive, more efficient and controlled space for robot learning. Embodied AI startup Hillbot is boosting its data pipeline by using Cosmos to generate terabytes of high-fidelity 3D environments. This AI-generated data will help the company refine its robotic training and operations, enabling faster, more efficient robotic skilling and improved performance for industrial and domestic tasks.
In both industries, developers can use NVIDIA Omniverse and Cosmos as a multiverse simulation engine, allowing a physical AI policy model to simulate every possible future path it could take to execute a particular task — which in turn helps the model select the best of these paths.
Data curation and the training of Cosmos models relied on thousands of NVIDIA GPUs through
NVIDIA DGX Cloud
, a high-performance, fully managed AI platform that provides accelerated computing clusters in every leading cloud.
Developers adopting Cosmos can use DGX Cloud for an easy way to deploy Cosmos models, with further support available through the
NVIDIA AI Enterprise
software platform.
Customize and Deploy With NVIDIA Cosmos
In addition to foundation models, the
Cosmos platform
includes a data processing and curation pipeline powered by
NVIDIA NeMo Curator
and optimized for NVIDIA data center GPUs.
Robotics and AV developers collect millions or billions of hours of real-world recorded video, resulting in petabytes of data. Cosmos enables developers to process 20 million hours of data in just 40 days on
NVIDIA Hopper GPUs
, or as little as 14 days on
NVIDIA Blackwell GPUs
. Using unoptimized pipelines running on a CPU system with equivalent power consumption, processing the same amount of data would take over three years.
The platform also features a suite of powerful video and image tokenizers that can convert videos into tokens at different video compression ratios for training various
transformer models
.
The Cosmos tokenizers deliver 8x more total compression than state-of-the-art methods and 12x faster processing speed, which offers superior quality and reduced computational costs in both training and
inference
. Developers can access these tokenizers, available under NVIDIA’s open model license, via
Hugging Face
and
GitHub
.
Developers using Cosmos can also harness model training and fine-tuning capabilities offered by
NeMo framework
, a GPU-accelerated framework that enables high-throughput AI training.
Developing Safe, Responsible AI Models
Now available to developers under the NVIDIA Open Model License Agreement, Cosmos was developed in line with NVIDIA’s
trustworthy AI
principles, which include nondiscrimination, privacy, safety, security and transparency.
The Cosmos platform includes Cosmos Guardrails, a dedicated suite of models that, among other capabilities, mitigates harmful text and image inputs during preprocessing and screens generated videos during postprocessing for safety. Developers can further enhance these guardrails for their custom applications.
Cosmos models on the
NVIDIA API catalog
also feature an inbuilt watermarking system that enables identification of AI-generated sequences.
NVIDIA Cosmos was developed by
NVIDIA Research
. Read the research paper, “
Cosmos World Foundation Model Platform for Physical AI
,” for more details on model development and benchmarks. Model cards providing additional information are available on
Hugging Face
.
Learn more about world foundation models in an
AI Podcast episode
that features Ming-Yu Liu, vice president of research at NVIDIA.
Get started
with NVIDIA
Cosmos
and join
NVIDIA at CES
. Watch the
Cosmos demo
and Huang’s keynote below:
See
notice
regarding software product information.
Categories:
Driving
|
Generative AI
|
Robotics
|
Software
Tags:
Artificial Intelligence
|
CES 2025
|
Cosmos
|
DGX Cloud
|
Jetson
|
NVIDIA DRIVE
|
NVIDIA NeMo
|
NVIDIA Research
|
Omniverse
|
Physical AI
|
Robotics
|
Simulation and Design
|
Synthetic Data Generation
|
Transportation
加速開發
實體人工智慧（AI）
的
NVIDIA Cosmos
平台推出一系列
世界基礎模型
，這是可以預測和產生虛擬環境未來狀態的物理感知影片神經網路，以協助開發人員打造下一代機器人和自動駕駛車。
世界基礎模型（WFM）與大型語言模型一樣都是最基本的模型。它們使用文字、圖像、影片和動作這些輸入資料來產生和模擬虛擬世界，以精準模擬場景中物體的空間關係及其實體互動的情況。
NVIDIA
今日在 CES 大會上宣布
推出第一波 Cosmos WFM，用於基於物理的模擬及產生合成資料，以及最先進的標記器（tokenizer）、護欄、加速資料處理與整理管道，以及模型客製化與最佳化框架。
不論其公司規模大小，都可以在 NVIDIA 允許商業用途的寬容式開放模型授權下，讓研究人員與開發人員自由使用 Cosmos 模型。建立 AI 代理的企業也可以使用 NVIDIA 在 CES 大會上發表的全新開放式
NVIDIA Llama Nemotron 和 Cosmos Nemotron 模型
。
Cosmos 最先進模型的開放性，排除建立機器人與自動駕駛車技術的
實體 AI
開發人員所面臨的障礙，讓各種規模的企業都能更快速地將其實體 AI 應用推向市場。開發人員可以直接使用 Cosmos 模型來產生基於物理的合成資料，也可以利用
NVIDIA NeMo 架構
，針對特定的實體 AI 設定，使用自己的影片來微調模型。
機器人公司 1X、Agility Robotics 與小鵬汽車，以及自動駕駛車開發商 Uber 及 Waabi 等實體 AI 領導廠商，都已經使用 Cosmos 加速和加強模型開發作業。
開發人員可以在
NVIDIA API 目錄
預覽第一批 Cosmos
自我回歸
和
擴散
模型，以及從
NVIDIA NGC 目錄
和
Hugging Face
下載一系列模型和微調框架。
實體
AI
的世界基礎模型
Cosmos 世界基礎模型是一套開放式擴散和自我回歸 transformer 模型，用於產生物理感知影片內容。使用 2,000 萬個小時現實世界人類互動、環境、工業、機器人和駕駛資料的 9,000 兆個詞元來訓練這些模型。
此模型有三個類別：Nano 適用於針對即時、
低延遲推論
與邊緣部署進行最佳化的模型；Super 適用於高效能基準模型；Ultra 適用於最高品質與真實度，最適合用於提取客製化模型。
搭配
NVIDIA Omniverse
3D 輸出內容使用時，擴散模型會產生可控制的高品質合成影片資料，以開始訓練機器人與自動駕駛車感知模型。自我回歸模型會根據輸入畫面和文字預測影片畫面序列中的下一個畫面。這樣就能即時預測下一個詞元，讓實體 AI 模型能夠預測它的下一個最佳動作。
開發人員可以使用 Cosmos 的開放模型來產生文字到世界和影片到世界的內容。擴散模型與自我回歸模型的版本各擁有 40 億到 140 億個參數，現在在 NGC 目錄與
Hugging Face
開放使用。
還有 120 億個參數的上採樣模型，用於細化文字提示；70 億個參數的影片解碼器，針對擴增實境進行最佳化；以及護欄以確保安全、負責任的使用 AI。
NVIDIA 也推出針對垂直應用的微調模型樣本，例如為自動駕駛車生成多感測器視角，以展示客製化的機會。
推動機器人及自動駕駛車技術的應用
Cosmos 世界基礎模型能夠
產生合成資料
以增強訓練資料集、先行模擬以在真實世界部署前對實體 AI 模型進行測試與除錯，以及在虛擬環境中進行強化學習以加速
AI 代理學習
。
開發人員可以使用 NVIDIA Omniverse 的 3D 合成場景來訓練 Cosmos，產生大量可控制、基於物理的合成資料。
從自駕車開始為實體世界開創生成式 AI 的 Waabi，正在評估使用 Cosmos 搜尋和整理影片資料，用於開發和模擬自動駕駛車軟體。這將進一步加速公司以業界領先的方式推動安全性的發展。該公司利用 Waabi World 這個生成式 AI 模擬器創建任何車輛可能遇到的情境，並以與真實世界相同的真實感呈現。
開發機器人的 WFM 可以產生合成的虛擬環境或世界，為機器人學習提供成本更低、更有效率且可控制的空間。體現 AI 新創公司 Hillbot 使用 Cosmos 來產生 TB 等級真實感十足的 3D 環境，以增強其資料管道。這些由 AI 產生的資料將有助於該公司完善其機器人訓練與操作，讓機器人更快、更有效率地學習各項技能，以及提高執行工業與家庭任務的表現。
這兩個產業的開發人員都可以使用 NVIDIA Omniverse 與 Cosmos 做為多重宇宙模擬引擎，讓實體 AI 策略模型模擬未來執行特定任務時可能採取的每個路徑，這反過來又能幫助模型從這些路徑中選擇最佳路徑。
Cosmos 模型整理資料和訓練必須依賴
NVIDIA DGX Cloud
平台上的數千個 NVIDIA GPU，而 NVIDIA DGX Cloud 是一個高效能、完全託管的 AI 平台，可在各大雲端環境提供加速運算叢集。
採用 Cosmos 的開發人員可以使用 DGX Cloud 輕鬆部署 Cosmos 模型，並且透過
NVIDIA AI Enterprise
軟體平台提供更多支援。
使用
NVIDIA Cosmos
進行客製化與部署
除了基礎模型之外，
Cosmos
平台
還有由
NVIDIA NeMo Curator
支援的資料處理與整理管道，並且針對 NVIDIA 資料中心 GPU 進行最佳化。
機器人與自動駕車開發人員收集數百萬或數十億小時的真實世界影片畫面，產生出 PB 等級的大量資料。Cosmos 讓使用
NVIDIA Hopper GPU
的開發人員，只要 40 天就能處理完 2,000 萬個小時的資料，而使用
NVIDIA Blackwell GPU
的話更只要 14 天。如果使用在 CPU 系統上執行的未最佳化管道作業，且功耗相當，則處理相同數量的資料則要三年以上的時間。
此平台還擁有一套功能強大的影片和圖像標記器，可以用不同的影片壓縮比將影片轉換為標記，用於訓練各種
transformer 模型
。
Cosmos 標記器的總壓縮率比最先進的方法高出 8 倍，處理速度高出 12 倍，在訓練和
推論
方面都能提供優異品質與降低運算成本。開發人員可以在
Hugging Face
及
GitHub
取得這些以 NVIDIA 開放模型授權提供的標記器。
使用 Cosmos 的開發人員也能利用
NeMo 框架
提供的模型訓練與微調功能，NeMo 框架是一個 GPU 加速框架，能夠以高處理量的方式來訓練 AI。
開發安全、負責任的
AI
模型
Cosmos現已根據 NVIDIA 開放模型授權協議提供給開發人員使用。Cosmos在開發的過程中遵照 NVIDIA
值得信賴的 AI
原則，包括公平性、隱私性、安全、保障與公開透明度。
Cosmos 平台包含一套專用的 Cosmos Guardrails 模型，它除了其他功能，還能在預先處理過程中減緩有害的文字與圖像輸入，並且在後製處理過程中篩選所產生的影片內容以確保安全性。開發人員可針對自訂應用進一步強化這些防護措施。
NVIDIA API 目錄
上的 Cosmos 模型另有內建浮水印系統，能夠發現 AI 產生的連續畫面。
NVIDIA Cosmos 由
NVIDIA Research
開發。請閱讀研究論文《
Cosmos World Foundation Model Platform for Physical AI
》，以瞭解更多關於模型開發與基準測試的詳細資訊。在
Hugging Face
有提供其他資訊的模型卡。
在 1 月 7 日播出的
AI Podcast
節目中，NVIDIA 研究部門副總裁 Ming-Yu Liu 將介紹更多關於世界基礎模型的資訊。
開始使用
NVIDIA Cosmos 並參加
NVIDIA 在 CES 大會的各項活動
。
﻿
請見有關軟體產品資訊的
通知
。
Categories:
生成式人工智慧
|
自主機器
|
自動駕駛
|
軟體
Tags:
Artificial Intelligence
|
CES 2025
|
Cosmos
|
DGX Cloud
|
Jetson
|
NVIDIA DRIVE
|
NVIDIA NeMo
|
NVIDIA Research
|
Omniverse
|
Robotics
|
Simulation and Design
|
Synthetic Data Generation
|
Transportation