REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

This post introduces REFACTOR-VLA, a system for unsupervised learning of reusable skills in VLA models. It proposes a 'wake/sleep' architecture utilizing a Behavioral-Equivalence Kernel (BEK) based on a learned latent world model for clustering action segments into skills. These skills are then represented as typed lambda terms and used by a library-conditioned rectified-flow action decoder. Skill acceptance is governed by Minimum Description Length (MDL) and return-preservation criteria. The training involves a three-phase schedule: world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. Key findings include that increasing world model size can degrade performance and that an auxiliary supervised contrastive loss during Phase A significantly improves skill clustering quality in Phase C, as measured by Normalized Mutual Information (NMI) on various LIBERO benchmark suites.