ModelExpress: Distributing Model Artifacts at the Speed of Light | NVIDIA Technical Blog

What this post added

Introduces ModelExpress (MX), a system for accelerating the distribution of model artifacts. MX prioritizes direct GPU-to-GPU P2P RDMA transfers via NIXL, bypassing object storage and host memory. It optimizes cold starts by streaming from object storage without landing on disk, using multithreaded tensor readers and pipelining reads with GPU placement. For clusters with shared disk caches, MX uses a Model Cache Service to coordinate downloads, ensuring the fleet populates the cache only once. When GPUDirect Storage (GDS) is supported, MX reads checkpoint files directly from local storage into GPU memory through NIXL's multithreaded GDS backend, bypassing host memory. MX also supports loading local checkpoints through ModelStreamer, overlapping disk I/O with GPU placement. For subsequent workers, MX treats serving replicas as live weight sources, transferring tensors directly from source GPU to target GPU over P2P RDMA via NIXL. The control plane discovers peers through Redis, Kubernetes CRDs, or k8s-service metadata. MX also optimizes NIXL memory registration overhead.