TechPrufer
Sign inGet started

Blogs›Together AI›Autoscaling for LLM Inference Endpoints

Autoscaling for LLM Inference Endpoints

Autoscaling for LLM Inference Endpoints

5
posts
2026

Autoscaling for LLM inference endpoints on Together AI allows users to configure deployments to automatically scale based on inference-native metrics such as in-flight requests, TTFT, GPU utilization, and token throughput. Users can set replica bounds, choose a scaling metric and target, and tune scale-up and scale-down windows to manage traffic fluctuations and optimize for latency and cost. The system supports various scaling metrics including concurrency-driven, SLO-driven, and efficiency-driven. This post details a benchmark comparing Together Inference Engine against TensorRT-LLM and SGLang on a production coding agent workload, demonstrating significant improvements in TPS and TTFT under high concurrency and long context scenarios. It highlights full-stack optimization techniques like ThunderMLA and custom kernel rewrites, and discusses the importance of production-relevant benchmarks that consider factors like KV cache pressure and prefill-heavy output shapes. The post also touches on the cost-effectiveness of their models compared to proprietary alternatives.

2026

Autoscaling endpoints for LLM inference

7/31/2026

This post introduces and details the autoscaling capabilities for LLM inference endpoints on Together AI. It explains the challenges of autoscaling LLM workloads compared to traditional stateless services, focusing on the nuances of GPU utilization metrics and the significant cost of cold starts. The post outlines the autoscaling policy configuration, including replica bounds, scaling metrics (e.g., in-flight requests, TTFT, GPU utilization), targets, and timing windows (`scale_up_window`, `scale_down_window`). It provides a catalog of available scaling metrics with guidance on choosing the appropriate one based on desired outcomes (concurrency, SLO, efficiency). It also addresses edge cases like rapid spikes, rollouts, traffic splitting, and the implications of idle shutdown and cold start durations.

Configuring Dedicated Model Inference

7/29/2026

Introduced the architecture for dedicated model inference, comprising endpoints, deployments, and configs. Detailed the capacity-aware traffic split mechanism, where traffic is routed proportionally to a deployment's effective capacity (weight * ready_replicas). Explained how this enables seamless autoscaling composition, controlled per-replica load, and robust handling of non-ready replicas. Clarified the distinctions between traffic splits, A/B experiment percents, and canary step percentages. Provided an experiment demonstrating capacity-aware routing and a comparison of latency vs. throughput profiles for different configurations.

ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

7/29/2026

Introduces ThunderAgent, a system that enhances agentic inference by treating agent workflows as schedulable programs rather than individual requests. This program-level abstraction addresses KV cache thrashing by selectively pausing low-priority workflows under memory pressure and routing resumed workflows to nodes with available capacity. ThunderAgent improves single-node throughput by up to 2.5x and achieves near-linear scaling on multi-node clusters, offering a drop-in solution compatible with existing inference engines and optimizations.

Benchmarking inference at scale: coding agents

5/19/2026

This post introduces a detailed benchmark for a production coding agent workload, comparing Together Inference Engine against TensorRT-LLM and SGLang. It highlights the engine's superior performance in TPS and TTFT under high concurrency and long context, attributing these gains to full-stack optimization including ThunderMLA (a fused multi-head latent attention kernel) and custom kernel rewrites. The benchmark methodology emphasizes realistic production conditions, such as concurrent requests, long prompts, and prefill-heavy output shapes, and discusses the degradation curves of different engines under load. It also presents cost savings achieved with their models compared to alternatives like Claude Opus.

Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

3/4/2026

Introduced Cache-Aware Prefill-Decode Disaggregation (CPD) to improve long-context LLM serving. CPD separates cold and warm workloads by cache hit rate, utilizing a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache via RDMA) and a router that steers requests to dedicated pre-prefill, prefill, or decode nodes. This architecture achieves up to 40% higher sustainable throughput and lower TTFT by preventing long-running cold prefills from blocking cache-reusing warm requests.