TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Tenant Isolation for Kubernetes on Shared GPU Infrastructure

4 posts · Jun 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of enabling isolated tenant Kubernetes clusters on shared GPU infrastructure. Initial efforts focused on addressing the challenges of managing multiple teams with conflicting requirements (CRDs, RBAC, Kubeflow versions) on shared hardware. The solution combines KAI Scheduler for topology-aware, hierarchical GPU scheduling with per-team quotas and dynamic allocation, and vCluster for provisioning fully isolated tenant clusters with independent control planes. Subsequent developments have introduced NodeWright for declarative, workload-aware host operating system management, enabling safe, progressive rollouts of system-level changes (kernel tuning, CVE remediation, security agent installation) across fleets without disrupting running workloads. NodeWright integrates with NVIDIA AI Cluster Runtime and DSX OS, providing a robust mechanism for managing the underlying infrastructure of shared GPU clusters.

All NVIDIA trails
  1. Manage Kubernetes Node Fleets with NodeWright | NVIDIA Technical BlogThis post introduces NodeWright, an open-source, Kubernetes-native package manager for declaratively configuring and safely updating host operating systems across GPU clusters. It orchestrates a six-stage sequence (cordon, wait, drain, apply, interrupt, uncordon) to manage host-level changes like kernel parameter tuning and CVE remediation without disrupting workloads, respecting PodDisruptionBudgets and non-interruptible workload labels. NodeWright's DeploymentPolicy resource enables progressive rollouts with configurable strategies and thresholds. It integrates with NVIDIA AI Cluster Runtime and DSX OS, providing a controlled path from provisioned to schedulable nodes.
  2. How to Carry User Identity Across Federated Kubernetes and AI Platforms | NVIDIA Technical BlogIntroduced a central identity gateway pattern for propagating user identity across federated Kubernetes and AI platforms. This pattern separates session ownership from request enforcement, allowing regional gateways to validate user identity through a shared session store instead of managing independent OIDC flows. This approach reduced repeated login events by 55% across internal developer platforms spanning Kubernetes clusters in AWS and OCI, while enabling unified platform shells and AI assistants with delegated user identity. The architecture uses standard OpenID Connect, a minimal /gateway/userinfo validation endpoint, stateless regional gateways, and a shared Redis-backed session store with explicit TTLs. Centralized session ownership coordinates token refresh and logout platform-wide, reduces upstream identity-provider load to scale with active users rather than user-tool combinations, and standardizes identity headers for downstream services. Security guardrails include mutual TLS or workload identity between gateways, stripping inbound identity headers before injecting trusted ones, and defining explicit failure behavior for session-store unavailability.
  3. How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure | NVIDIA Technical BlogThis post details the practical implementation of running isolated tenant Kubernetes clusters on shared GPU infrastructure by combining KAI Scheduler and vCluster. It provides a step-by-step tutorial for setting up a demo environment with one NVIDIA L40S GPU shared among three teams (NLP, Vision, Recommender). Key contributions include: configuring KAI Scheduler for topology-aware, hierarchical GPU scheduling with per-team quotas and dynamic allocation via custom queue CRDs; utilizing vCluster's shared-nodes model to provision fully isolated tenant clusters with independent control planes, RBAC, CRDs, and cluster-admin access; and demonstrating how these components work together to ensure logical separation while maximizing infrastructure efficiency. The post also covers local tooling setup, Helm repository addition, GPU Operator confirmation, and the installation of KAI Scheduler and vCluster.
  4. One-Click Multi-Tenant Security with NVIDIA Quantum InfiniBand | NVIDIA Technical BlogIntroduced intent-based security profiles in Unified Fabric Manager (UFM) for NVIDIA Quantum InfiniBand, enabling one-click multi-tenant fabric security and tenant isolation. This includes three predefined profiles (General, Bare Metal Cloud, Secured Bare Metal Cloud) that automate PKey isolation, MAD key protection, and GUID-based access control. Also introduced Continuous Security Verification (CSV) for automated auditing and remediation of security posture.