Decoupled DiLoCo: Resilient, Distributed AI Training at Scale

Introduces Decoupled DiLoCo, a distributed AI training architecture that divides training runs into decoupled 'islands' of compute with asynchronous data flow. This architecture isolates local disruptions, allowing other parts of the system to continue learning efficiently, leading to greater hardware resiliency and lower bandwidth requirements. It also enables mixing different hardware generations (e.g., TPU v6e and TPU v5p) in a single training run.