TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Instant GPU Clusters

12 posts · Sep 2025 – Oct 2026

What this trail covers

Together GPU Clusters now integrate autoscaling, Role-Based Access Control (RBAC), full-stack observability, self-serve node repair, and active health checks. Autoscaling leverages Kubernetes Cluster Autoscaler for elastic capacity. Health checks and self-repair reduce Mean Time To Repair (MTTR) for hardware failures. RBAC provides structured multi-team governance through projects and roles (Admin, Member). Full-stack observability is provided via dedicated Grafana instances with pre-built dashb. This collaboration expands enterprise-grade inference capacity by deploying large clusters of NVIDIA B300 GPUs on IBM Cloud, leveraging NVIDIA Spectrum-X Ethernet networking for high-throughput inference.

All Together AI trails
  1. Expanding our enterprise inference capacity with IBM Cloud and NVIDIAThis post details the deployment of a dedicated NVIDIA B300 GPU cluster on IBM Cloud, specifically for AI inference. It highlights the collaboration between Together AI (inference platform), IBM Cloud (infrastructure), and NVIDIA (silicon and networking). The focus is on scaling enterprise-grade AI inference to meet rapidly growing token demand, emphasizing the benefits of open models for enterprises (data sovereignty, cost-effectiveness) and the need for production-grade reliability, security, and guardrails at massive scale.
  2. Introducing preemptible compute: the same compute, half the priceIntroduces preemptible compute as a new compute type for Together GPU Clusters, available on Kubernetes. This feature offers a flat 50% discount on on-demand rates for interruption-tolerant workloads. It details the five-minute drain sequence upon reclamation, sub-hourly billing, and how preemptible nodes are labeled and scheduled. The post provides guidance on suitable workloads (short experiments, inference bursts, batch jobs) and critical considerations before enabling, such as treating nodes as ephemeral and keeping critical components on standard nodes. It also outlines the setup process for new and existing clusters.
  3. New in Together GPU Clusters: Reliability and control for production GPU clustersIntroduced passive health checks to detect failures during active workloads, complementing existing active checks. Implemented auto node repair with human-in-the-loop approval for actions like reboot, reprovision, failover, and remove. Upgraded the Slurm-on-Kubernetes stack (Slinky 2.0) with self-healing worker daemons, no zombie processes, durable job accounting, reliable process cleanup, and accurate GPU state after reschedules. Enhanced operational control with a new cluster details view (health, usage, event timeline), external OIDC for Kubernetes RBAC, and startup scripts for cluster customization at boot, job start, and job end.
  4. Capacity without conflict: A guide to multi-tenant GPU cluster design for AI-native teamsThis post details the design principles, common failure modes, and implementation of multi-tenant GPU cluster architecture, focusing on pooled capacity, tenant isolation, and self-serve access. It outlines the two-layer infrastructure pattern (shared foundation, per-tenant top layer), quota-based allocation for capacity management, flexible configuration options for tenants, and robust GPU health check/node repair mechanisms. The goal is to provide data center unit economics with cloud-like flexibility and self-service velocity for AI-native teams.
  5. What is an AI Native Cloud?This post defines the concept of an 'AI Native Cloud' and outlines its key characteristics: a full AI stack from hardware to software, a fast path from research to production, reliability at massive scale, a builder-centric approach, and a partnership model that moves at AI-native pace. It positions Together AI as building this AI Native Cloud, emphasizing its vertical integration and continuous evolution with research innovations. While it references existing infrastructure like GPU clusters, its primary contribution is conceptualizing and defining the overarching cloud strategy for AI-native companies.
  7. New in Together GPU Clusters: Autoscaling, observability, and self-healingThis post details the integration of autoscaling (powered by Kubernetes Cluster Autoscaler), RBAC with project-based roles (Admin, Member), active health checks and self-serve node repair for reducing MTTR, and full-stack observability via Grafana dashboards into Together GPU Clusters. It explains how these features enable elastic capacity, improve reliability by reducing failure impact, provide structured governance, and offer deep telemetry for debugging and performance tuning.
  8. Key research and product announcements at the AI Native ConfThis post introduces several significant technical advancements that enhance AI infrastructure and performance. Key contributions include: FlashAttention-4, a new attention algorithm optimized for NVIDIA Blackwell GPUs, offering substantial speedups over existing solutions. Together Megakernel, a single-kernel implementation for running entire models, achieving significant latency reduction for real-time voice agents. together.compile, an automation tool for generating optimized GPU kernels, improving generation speed for video and image models. The Reinforcement Learning API is enhanced with infrastructure for RL training, focusing on efficient weight distribution and rollout optimization. ThunderAgent is introduced as a program-aware abstraction for agentic workloads, addressing KV cache thrashing, cross-node memory imbalance, and tool lifecycle issues, leading to improved throughput and memory savings. ATLAS-2 provides an online training flywheel for speculative decoding, continuously updating speculators from live traffic for sustained performance gains. Cache-aware prefill–decode disaggregation (CPD) introduces a three-tier serving stack to optimize long-context inference by intelligently routing requests based on cache hit rates, significantly increasing sustainable throughput.
  9. Inside multi-node training: How to scale model training across GPU clustersThis post details the infrastructure, techniques, and practical steps for scaling model training across GPU clusters, focusing on multi-node distributed training. It explains parallelism strategies (data, tensor, pipeline), network interconnects (NVLink, InfiniBand), and checkpointing/fault tolerance. A production example of training Qwen2.5-72B is provided, along with guidance on getting started, infrastructure verification, framework configuration, checkpointing implementation, scaling tests, and monitoring. It also covers common failure modes and troubleshooting for multi-node training.
  10. How to run TorchForge reinforcement learning pipelines in the Together AI Native CloudThis post details the integration of TorchForge reinforcement learning (RL) pipelines with Together AI's Instant Clusters. It highlights the use of Instant Clusters' low-latency GPU communication, preconfigured drivers, and heterogeneous scheduling for RL workloads. The post also describes the integration of Together CodeSandbox and Together Code Interpreter as environments for RL agents, enabling tool-use and code execution. A demo is provided for training a BlackJack agent using GRPO, vLLM, and TorchStore, and a standalone integration for using Code Interpreter as an OpenEnv environment is also released.
  11. Announcing General Availability of Together Instant Clusters, offering ready to use, self-service NVIDIA GPUsThis post announces the General Availability of Together Instant Clusters, a self-service, API-first offering for provisioning multi-node GPU clusters in minutes. It details the integration of NVIDIA Hopper and Blackwell GPUs, pre-configured orchestration (Kubernetes/Slurm), and high-performance networking (InfiniBand, Spectrum-X Ethernet with RoCE). The post highlights optimizations for distributed training (NVLink, NVLink Switch) and scalable burst capacity for inference, emphasizing reliability through node testing, continuous monitoring, and SLAs. It also mentions features like driver/CUDA version pinning, BYO container images, and shared storage.