NVIDIA Cosmos 3: Robots finally take over

This post introduces NVIDIA Cosmos 3, a world foundation model for physical AI, and details its deployment on Baseten. It explains two primary usage patterns for robotics: direct control via the `policy` mode and data generation for training smaller onboard models using `inverse_dynamics`, `forward_dynamics`, `text2video`, and `image2video` modes. The post highlights Cosmos 3's ability to generate physically plausible data, contrasting it with creative video models, and emphasizes its role in overcoming the data collection bottleneck in robotics by transforming real-world or synthetic video into labeled training data. It also provides example API requests for `policy` and `inverse_dynamics` modes, showcasing parameters like `model_mode`, `vision_path`, `action_chunk_size`, and `raw_action_dim`. The post notes that Cosmos 3 Nano can be deployed on Baseten with specific cold-start and warm inference times.