TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Slurm Topology-Aware Job Scheduling for GB200 NVL72

2 posts · May 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of job scheduling for large-scale AI and HPC clusters, focusing on optimizing resource utilization and performance by understanding the underlying hardware topology. Initial efforts focused on traditional scheduling approaches. Subsequent developments have introduced topology-aware scheduling plugins for workload managers like Slurm, specifically designed for modern architectures like NVIDIA GB200 NVL72. This includes the development of topology/block plugins and tools like NVIDIA Topograph to discover and publish cluster topology information to schedulers, enabling efficient workload placement within locality domains and across systems and racks. This ensures GPUs are placed in the most efficient locations, minimizing contention and latency, and maximizing throughput and resource utilization.

All NVIDIA trails
  1. Topology-Aware Workload Scheduling with NVIDIA Topograph | NVIDIA Technical BlogThis post introduces NVIDIA Topograph, an open-source toolkit that discovers cluster topology from cloud APIs or on-premises fabric systems and normalizes it into a common model. It publishes this topology data in formats usable by workload managers like Kubernetes (node labels, NFD resources), Slurm (topology configuration), and Slinky (ConfigMaps). Topograph's architecture includes an API Server, Node Observer, Node Data Broker, Providers (e.g., for InfiniBand, Spectrum-X, cloud APIs), and Engines. It addresses the challenge of keeping scheduler topology views current by regenerating them on request and upon watched cluster changes. The post details deployment options and integration with schedulers like KAI Scheduler for topology-aware gang scheduling.
  2. Unlock Exascale Performance on NVIDIA GB200 NVL72 with Slurm Topology-Aware Job Scheduling | NVIDIA Technical BlogThis post details the implementation and benefits of Slurm's new topology/block plugin (co-developed with SchedMD for Slurm 23.11) for NVIDIA GB200 NVL72 systems. It explains how this plugin enables topology-aware job scheduling by aligning jobs with NVL72 domain boundaries, minimizing fragmentation, and optimizing GPU occupancy. The post highlights the concept of job segment sizes, which define nodes communicating over NVLink, and recommends best practices for segment sizing (larger for MoE and high I/O, smaller for general jobs) based on simulation results showing near-theoretical maximum GPU occupancy. It also emphasizes continuous monitoring of fragmentation and segment size adjustment for sustained performance.