Intro to NVIDIA Cosmos with Ming-Yu ft. Superintelligence | Cosmos Labs

Real-world data collection is prohibitively expensive

Collecting sufficient training data for physical AI is slow, costly, and never comprehensive enough to cover the diverse, unpredictable nature of the physical world.

Visual world requires richer representation than language

Unlike LLMs trained on internet text, physical AI demands world foundation models because visual-motor skills are difficult to describe in language but easy to model through pixels and physics.

Cosmos Predict generates future world states

This base model creates video predictions from current images and actions, enabling trajectory forecasting and interactive closed-loop simulations where the world responds to robot actions.

Cosmos Transfer closes the sim-to-real gap

It transforms outputs from physics simulators like Isaac Sim into photorealistic video while maintaining Newtonian physics accuracy, and can augment demonstrations across diverse environments.

Cosmos Reason provides video evaluation

Acting as a visual language model, it analyzes whether tasks are completed successfully, assigns reward signals for training, and breaks down edge cases into familiar physical interactions.

Video-native policies outperform VLM-based approaches

Cosmos Policy is built atop video models rather than visual language models, enabling more precise prediction of pixel-level dynamics crucial for robot control.

Model predictive control through value functions

The system simulates multiple possible action sequences, assigns values to predicted outcomes, and selects optimal trajectories similar to having a chess engine for physical tasks.

Training on worlds enables interactive learning

Unlike passive data consumption, this approach allows models to learn from customized interactive experiences where actions change world states and generate environmental feedback.

Physical AI requires hardware customization

Because robots have diverse sensor configurations ranging from three to seven cameras with various LiDAR setups, open weights and architecture are essential for developers to configure models to their specific hardware.

Domain specialization through post-training

Developers can specialize the 8B parameter models for specific environments like warehouses or manufacturing lines by post-training on targeted data rather than relying solely on zero-shot prompting.