Optimizing delta weight syncs for managed rollouts

This post details Baseten's implementation of delta weight synchronization for managed rollouts of large AI models. It describes a system that allows trainers to publish checkpoints or deltas to object storage, which are then distributed to independent rollout clusters via the Baseten Delivery Network. A key technical innovation is the custom zstd decoder that efficiently processes XOR-encoded weight differences, avoiding the materialization and scanning of the full model. This sparse decoding approach, combined with optimized GPU update strategies that group changes by destination and serialize updates for retry safety, significantly reduces the request pause time during model updates from ~12 seconds to ~6 seconds, achieving full fleet updates in under 40 seconds.