From tokens to concepts: how particle models perceive the world

This post introduces the extension of Deep Latent Particles (DLPs) to 3D scenes, enabling self-supervised decomposition of colored 3D observations into object-centric particles. It details the replacement of the keypoint-proposal mechanism with an appearance-aware K-means prior for clustering in a joint color-and-position space and the implementation of a chroma loss to address color reconstruction issues. The work demonstrates that these 3D particles can drive diffusion-based manipulation policies, outperforming baselines that ignore 3D geometry or use dense voxel inputs.