How Claude Performs on Robotics Tasks

This post details Anthropic's evaluation of language models' performance in robotics across various control interfaces and robot bodies. It highlights that model performance is highly dependent on the control interface, with models struggling with direct low-level torque control but showing significant gains when supervising pretrained policies or using high-level commands. The research quantifies generational improvements in locomotion and manipulation tasks, noting that newer models are better at adapting strategies and converting sensory input into actions. The post also identifies limitations, such as difficulties with stable spatial memory and long open-loop plans, and discusses the implications for safe AI development.