TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

World Action Models for Robot Manipulation

8 posts · Jun 2026 – Aug 2026

What this trail covers

This feature thread tracks the evolution of World Action Models (WAMs) for robot manipulation, focusing on their ability to generalize beyond training data by learning world dynamics. Initial efforts focused on the limitations of Vision-Language Models (VLMs) in robotics, which excel at semantic understanding but lack predictive dynamics. WAMs, built on video world models, overcome this by learning how the world evolves, enabling better physical generalization, adaptation to new robots with fewer training examples, and the development of NVIDIA Cosmos 3. Cosmos 3 unifies physical reasoning, world generation, and action generation within a single Mixture-of-Transformers (MoT) architecture, featuring a Reasoner tower (VLM) and a Generator tower (diffusion-based). It supports multiple modalities and offers Nano (16B) and Super (64B) model checkpoints, along with open datasets and deployment tools (Cosmos NIM microservices).

All NVIDIA trails
  1. Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science | NVIDIA Technical BlogThis post details the integration of the NVIDIA BioNeMo Agent Toolkit with Claude Science to orchestrate protein structure prediction workflows. It outlines the setup process for running BioNeMo NIM microservices (msa-search, OpenFold3, Boltz-2) on GPUs, including downloading databases and configuring compute endpoints. The post provides specific prompts for generating MSAs and predicting protein structures for monomers and heteromers, demonstrating how AI agents can leverage specialized domain expertise for complex scientific tasks. It also presents benchmark results comparing OpenFold3 and Boltz-2 for protein structure prediction with and without MSA input, highlighting the critical role of evolutionary alignment.
  2. How to Train a Cross-Embodiment Robot Navigation Policy with AI Agents | NVIDIA Technical BlogThis post details the application of the COMPASS framework and an agent-driven workflow to train a cross-embodiment robot navigation policy. It introduces the COMPASS framework, which adapts a pretrained X-Mobility policy into embodiment specialists using residual Reinforcement Learning (RL). The agent-driven workflow automates tasks such as environment validation, scene preparation, smoke testing, residual training, checkpoint evaluation, and runtime packaging, using Codex as the coding agent. The reference implementation uses the Boston Dynamics Spot quadruped and demonstrates training in a built-in warehouse scene, generated SAGE-10K scenes, and captured environments reconstructed with NVIDIA Omniverse NuRec. It also outlines the evaluation process comparing base and residual policies using COMPASS metrics and discusses runtime integration with optional cuVSLAM for odometry.
  3. Post-Train NVIDIA Cosmos 3 Edge for On-Device Robot Control | NVIDIA Technical BlogThis post details the post-training of NVIDIA Cosmos 3 Edge, a 4B omni-model, for on-device robot control on NVIDIA Jetson Thor. It describes the process of adapting a pre-trained world model to fit within device memory and achieve control latency requirements for real-time operation. The post outlines the training data (Cosmos3-DROID dataset), configuration, and deployment steps, including serving the policy via a WebSocket server on Jetson Thor. It demonstrates that the model can run entirely on the robot, generating actions in approximately 1.53 seconds per chunk at 640x540 resolution and 15 Hz, enabling continuous robot motion. The post also provides a smoke test for the policy server and discusses bringing custom robot data.
  4. Beyond VLAs: How World Action Models Reshape Robot Manipulation | NVIDIA Technical BlogThis post details how World Action Models (WAMs), built on video world models, overcome the limitations of Vision-Language Models (VLMs) in robotics by learning world dynamics for better physical generalization and adaptation. It introduces the open NVIDIA Cosmos 3 model, a Mixture-of-Transformers (MoT) omni-model trained on a large multimodal dataset, as a strong foundation for WAMs. The post also describes post-trained policies like Cosmos3-Nano-Policy-DROID and Cosmos3-Edge-Policy-DROID, highlighting their ability to predict future world states alongside actions, retain the full omni architecture, and demonstrate measurable improvements in policy performance (e.g., RoboLab success rate increase from 28.1% to 36.8% with an omni checkpoint). Deployment considerations for workstation serving and on-device inference are also discussed.
  5. Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super | NVIDIA Technical BlogIntroduces NVIDIA Alpamayo 2 Super, a 34B parameter reasoning vision-language-action (VLA) model that unifies trajectory generation, Chain-of-Causation (CoC) reasoning traces, meta-action prediction, VQA with 2D grounding, and auto-labeling for autonomous vehicle (AV) workflows. The model processes multi-camera video, language context, and ego-motion history to output these complementary outputs, enabling common foundation model use across offline policy teaching, evaluation, data curation, and task customization. It achieves state-of-the-art results on multiple AV benchmarks, including trajectory prediction (minADE_6 of 0.911 m), AV reasoning (0.433), and LingoQA (79.2).
  7. How to Evaluate General-Purpose Robot Policies for Real-World Deployment | NVIDIA Technical BlogThis post introduces RoboLab, a simulation benchmarking platform for general-purpose robot policies. It addresses issues of visual domain overlap, benchmark saturation, and diagnostic gaps in existing benchmarks. RoboLab enables robot-agnostic evaluations with rapid, scalable task and scene generation, and provides advanced diagnostic tools like graded task scores, trajectory quality analysis (SPARC), and detailed failure event logging. It also emphasizes statistical trustworthiness by discussing confidence intervals using the Clopper-Pearson method. The platform supports agentic AI workflows for generating novel tasks.
  8. Develop Humanoid Robot Policies End-to-End with NVIDIA Isaac GR00T | NVIDIA Technical BlogThis post introduces the NVIDIA Isaac GR00T Development Platform as an end-to-end solution for humanoid robot policy development. It details the GR00T 1.7 vision-language-action (VLA) model, which uses a Cosmos-Reason2-2B backbone and is pretrained on ~32K hours of real human demonstration and ~8K hours of simulation data. Key advancements include expanded deployment support via ONNX/TensorRT export, enhanced long-horizon task reasoning through task/subtask decomposition, and improved benchmarks on DROID and SimplerEnv. The post outlines a simulation workflow for a pick-and-place task, covering environment setup, data collection with Isaac Teleop, data conversion to LeRobot format, post-training GR00T 1.7, and policy evaluation in Isaac Lab Arena, emphasizing the use of AGILE WBC for static tasks and the role of joint-space targets as training signals.
  9. Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models | NVIDIA Technical BlogThis post analyzes the recent shift in robot foundation models from Vision-Language-Action (VLA) models to World-Action Models (WAMs). It hypothesizes that WAMs, by leveraging pretrained video or world-model backbones that already model scene dynamics, can better address the language-to-action grounding gap compared to VLM-based VLAs. The post categorizes modern WAMs based on what they predict (inverse dynamics, joint prediction, representation-only) and how actions are integrated (default action tokens, action as image, latent actions). It also discusses architectural considerations and the reasons for the recent popularity of WAMs, suggesting they will become a major recipe for robot foundation models, likely in hybrid forms.
  10. Develop Physical AI Reasoning, World, and Action Models with NVIDIA Cosmos 3 | NVIDIA Technical BlogThis post introduces NVIDIA Cosmos 3, a frontier foundation model for physical AI that unifies physical reasoning, world generation, and action generation. It features a Mixture-of-Transformers (MoT) architecture with a Reasoner tower (VLM) and a Generator tower (diffusion-based). Cosmos 3 supports multimodal inputs and outputs (text, image, video, action) and is available in Nano (16B) and Super (64B) parameter sizes. The release includes open-source models, training scripts, deployment tools (Cosmos NIM microservices), and synthetic datasets for robotics, autonomous driving, and warehouse environments.