Distributed AI Training Infrastructure

Decoupled DiLoCo is a new distributed AI training architecture that enables training LLMs across distant data centers with lower bandwidth and increased hardware resiliency. It divides training runs into decoupled 'islands' of compute with asynchronous data flow, isolating local disruptions and allowing other parts of the system to continue learning efficiently. This approach builds on Pathways and DiLoCo, offering fault-tolerant asynchronous training at scale. It has demonstrated greater availability of learning clusters than traditional methods and can mix different hardware generations in a single training run.