TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Production GPU Cluster Reliability and Control

1 post · Jul 2026

What this trail covers

Together GPU Clusters now offer enhanced reliability and operational control for production workloads. This includes passive health checks and auto node repair to catch and recover from hardware failures, a rebuilt Slurm-on-Kubernetes stack (Slinky 1.0) for improved worker daemon resilience, durable job accounting, reliable process cleanup, and accurate GPU state management after reschedules. Operational control features include a new cluster details view with health and usage metrics, external OIDC for Kubernetes RBAC for granular access control, and startup scripts for cluster customization at boot, job start, and job end.

All Together AI trails
  1. New in Together GPU Clusters: Reliability and control for production GPU clustersThis post introduces several new features for Together GPU Clusters focused on production reliability and operational control. Key contributions include: 1. Passive health checks that monitor real workloads for failures like GPU errors and thermal throttling, complementing existing active checks. 2. Auto node repair with a human-in-the-loop approach, offering actions like reboot, reprovision, failover, and remove based on detected issues. 3. Slinky 1.0, a rebuilt Slurm-on-Kubernetes stack, addressing issues like self-healing worker daemons, zombie process cleanup, durable job accounting, reliable process cleanup, and accurate GPU state after reschedules. 4. A new cluster details view providing at-a-glance health, usage, and event timelines. 5. External OIDC support for Kubernetes RBAC, enabling per-user authentication and authorization against external identity providers. 6. Startup scripts for Slurm worker nodes, login nodes, and controllers to automate node setup, job environment preparation, and cleanup tasks.