TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Nonuniform Tensor Parallelism for LLM Training

1 post · Jul 2026

What this trail covers

This feature thread tracks the evolution of Nonuniform Tensor Parallelism (NTP) for large-scale LLM training. NTP enables training jobs to dynamically adapt their tensor parallelism degree in response to transient GPU unavailability, ensuring sustained Goodput and minimizing computational stalls. It incorporates dynamic power boosting to compensate for resource reductions and uses efficient, overlapped tensor resharding techniques to limit overhead to less than 1%. NTP supports scale-up domains up to 72 GPUs on NVIDIA Blackwell and Blackwell Ultra systems with NVLink interconnects.

All NVIDIA trails
  1. Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading | NVIDIA Technical BlogThis post details the implementation and performance benefits of host offloading for JAX-based LLM training, specifically on NVIDIA Grace Blackwell and Vera Rubin platforms. It explains how moving selected activations to pinned host memory during the forward pass and streaming them back during the backward pass alleviates High-Bandwidth Memory (HBM) pressure. Experiments with MaxText on NVIDIA GB200 NVL72 using DeepSeek-V3 671B and Llama 3.1 405B demonstrated up to 57% throughput improvement over activation rematerialization and enabled larger batch sizes that would otherwise cause Out-of-Memory (OOM) errors. The post highlights the importance of overlapping activation transfers with compute and communication using XLA custom scheduling flags and dedicated copy streams for optimal performance, emphasizing the tight co-design of software and hardware on NVIDIA platforms.
  2. Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism | NVIDIA Technical BlogThis post details the implementation and benefits of Nonuniform Tensor Parallelism (NTP) for large-scale LLM training. It explains how NTP dynamically adjusts the tensor parallelism degree in response to transient GPU unavailability, preventing stalls and maintaining Goodput. The post also highlights the integration of dynamic power boosting to compensate for reduced GPU availability and describes efficient, overlapped tensor resharding techniques that minimize overhead to less than 1%. It showcases NTP's effectiveness on NVIDIA Blackwell and Blackwell Ultra systems with NVLink interconnects, supporting scale-up domains up to 72 GPUs.