TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

GPU Observability for Kubernetes

2 posts · May 2026 – Aug 2026

What this trail covers

This feature thread tracks the evolution of GPU observability solutions for Kubernetes environments. Initial efforts focused on addressing the gap in standard Kubernetes metrics for GPU utilization, leading to the development of tools like the GPU Usage Monitor. This project integrates DCGM Exporter, kube-state-metrics, Prometheus, and Grafana into a single, easy-to-deploy Helm chart. The goal is to provide real-time visibility into GPU allocation, compute utilization, memory consumption, and pod status across an entire Kubernetes cluster and through a single Helm chart deployment. The latest iteration focuses on operational simplicity, enabling actionable GPU visibility within minutes via a single Helm install command, complete with pre-built dashboards designed specifically for GPU-accelerated workloads.

All NVIDIA trails
  1. How to Choose Full-Stack Observability for NVIDIA AI Factories | NVIDIA Technical BlogThis post details a practical framework for choosing full-stack observability for NVIDIA AI factories, focusing on integrating telemetry across compute, networking, storage, orchestration, and application layers. It outlines a decision process for mapping AI infrastructure components to specific telemetry tools (DCGM, NVSM, UFM, NetQ, NMX, BCM, Run:ai, NIM) based on failure domains and component types. The post provides a concrete example of applying this framework to an InfiniBand cluster, emphasizing the selection of a minimal toolset (IPMI, DCGM, NVSM, UFM, BCM) unified in Prometheus/Grafana to detect fabric and GPU health regressions before significant job waste occurs. It also stresses the importance of building an actionable, top-k alert set tied to SLO/SLI metrics and avoiding alert fatigue.
  2. Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters | NVIDIA Technical BlogThis post details the GPU Usage Monitor, an open-source project that deploys a fully integrated GPU observability stack for Kubernetes. It leverages DCGM Exporter, kube-state-metrics, Prometheus, and Grafana, packaged into a single Helm chart for simplified deployment. The monitor provides real-time visibility into GPU allocation, compute utilization, memory consumption, and pod status, addressing common failure modes like over-provisioning and pod starvation. Key features include tracking namespaces and workloads, displaying per-GPU utilization against thresholds, breaking down memory consumption by pod, and showing running vs. pending pod counts. Configuration options for external Prometheus integration, custom resource allocation, and credential management are also highlighted.