Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading | NVIDIA Technical Blog

Introduces host offloading in JAX for LLM training to reduce HBM bottlenecks by moving activations to host memory. Demonstrates significant throughput improvements (up to 57%) and increased feasible batch sizes on NVIDIA GB200 NVL72 systems using MaxText with Llama 3.1 405B and DeepSeek-V3 671B models. Highlights the importance of co-design between XLA compiler and NVIDIA hardware (NVLink-C2C) for efficient asynchronous data movement and performance gains, especially for sparse MoE models.