Omni-bodied Learning from Video

4 posts · Aug 2025 – Jan 2026

What this trail covers

Skild AI's Omni-bodied Learning enables robots to acquire new skills by watching human videos, bridging the embodiment gap and overcoming the data bottleneck in robotics. This approach leverages abundant human video data and requires minimal robot-specific data for fine-tuning. This post details the development of an end-to-end locomotion control system driven entirely by online vision and proprioception, enabling humanoid robots to seamlessly walk across flat ground, climb stairs, and step over obstacles without explicit planning or behavior switching.

All Skild AI trails

Jan 2026Learning by watching human videosThis post elaborates on Skild AI's Omni-bodied Learning capability by detailing the technical challenges and solutions for learning robot skills from human video demonstrations. It highlights the 'data bottleneck' in robotics compared to LLMs, the limitations of teleoperation (diversity and scale), and the 'embodiment gap' (missing signals and morphological differences). The core contribution is the advancement of the Omni-bodied Learning model to learn new skills directly from video with minimal robot-specific data (<1 hr), making scalable task learning for foundation models feasible.
Sep 2025The case for an omni-bodied robot brainThis post details the engineering effort behind creating an 'omni-bodied brain' for robots, trained on a simulated universe of 100,000 different robot bodies. The core contribution is demonstrating how training on a diverse set of morphologies forces a model to develop adaptive locomotion strategies. The post highlights the model's ability to perform zero-shot control and adapt to extreme morphological changes (loss of limbs, broken legs, jammed wheels, walking on stilts) through in-context learning, adapting within milliseconds to minutes. It showcases how the model learns from failures by prepending previous trial data as a prompt, improving behavior over successive attempts. This contrasts with specialist controllers that fail catastrophically in novel situations. The technical novelty lies in the scale of simulated diversity and the observed emergent adaptability and resilience in physical robot control.
Aug 2025One Model, Any Scenario: End-to-end Locomotion from VisionThis post details the low-level control capabilities of Skild Brain, a general vision-based robot model. It introduces a single neural network that directly outputs low-level motor commands from raw images and joint feedback, enabling end-to-end locomotion control driven by online vision and proprioception. The model demonstrates dynamic adaptation to new terrain, including stairs and obstacles, without prior planning or mapping, and shows robustness to external forces and precision in foot placement, even when carrying payloads.
Advertisement