Blogs›NVIDIA›GPU Observability for Kubernetes

GPU Observability for Kubernetes

posts

2026

This feature thread tracks the evolution of GPU observability solutions for Kubernetes environments. Initial efforts focused on addressing the gap in standard Kubernetes metrics for GPU utilization, leading to the development of tools like the GPU Usage Monitor. This project integrates DCGM Exporter, kube-state-metrics, Prometheus, and Grafana into a single, easy-to-deploy Helm chart. The goal is to provide real-time visibility into GPU allocation, compute utilization, memory consumption, and pod status across clusters, enabling platform teams to optimize resource usage, identify scheduling bottlenecks, and prevent over-provisioning. Subsequent developments will likely focus on further integration with AI factory infrastructure, advanced alerting capabilities, and support for new GPU architectures and Kubernetes features.

2026

Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters | NVIDIA Technical Blog

5/21/2026

This post introduces the open-source GPU Usage Monitor, a project designed to provide real-time visibility into GPU usage across Kubernetes clusters. It details the observability gap in existing Kubernetes monitoring stacks for GPUs and presents a solution that integrates DCGM Exporter, kube-state-metrics, Prometheus, and Grafana via a single Helm chart. The post outlines the architecture, installation process, and key insights provided by the pre-built Grafana dashboards, including GPU allocation trends, compute utilization, memory usage per workload, and running/pending pod counts. It also discusses configuration options for integration with existing Prometheus instances and credential management.