Giving robots 3D vision without depth sensors

This post introduces StereoPolicy, a novel approach for robots to gain 3D perception from stereo images without explicit depth sensors. It replaces single-camera observations with synchronized stereo image pairs, processing them with pretrained 2D vision encoders and fusing features via a cross-attention Stereo Transformer. This method implicitly captures 3D structure by learning spatial correspondence and disparity. The post details two implementations: StereoPolicy-DP (trained from scratch with diffusion policies) and StereoPolicy-VLA (fine-tuning on vision-language-action models). It presents quantitative results showing StereoPolicy's superior success rates on real-world tasks compared to RGB-only, RGB-D, and point-cloud-based methods, especially for transparent objects and in low-data scenarios. The work highlights the broader trend towards physical AI and the increasing demand for GPU compute for training and deployment.