How to Choose Full-Stack Observability for NVIDIA AI Factories | NVIDIA Technical Blog

This post details a practical framework for choosing full-stack observability for NVIDIA AI factories, focusing on integrating telemetry across compute, networking, storage, orchestration, and application layers. It outlines a decision process for mapping AI infrastructure components to specific telemetry tools (DCGM, NVSM, UFM, NetQ, NMX, BCM, Run:ai, NIM) based on failure domains and component types. The post provides a concrete example of applying this framework to an InfiniBand cluster, emphasizing the selection of a minimal toolset (IPMI, DCGM, NVSM, UFM, BCM) unified in Prometheus/Grafana to detect fabric and GPU health regressions before significant job waste occurs. It also stresses the importance of building an actionable, top-k alert set tied to SLO/SLI metrics and avoiding alert fatigue.