TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Autoscaling for LLM Inference Endpoints

7 posts · Aug 2025 – Sep 2026

What this trail covers

Together AI offers advanced deployment strategies for LLM inference endpoints, including canary rollouts with metric gates for safe model upgrades, blue-green deployments for rapid switching, and rolling updates for in-place replica swaps. These strategies allow for controlled traffic migration, automated health and performance checks, and robust rollback mechanisms to ensure zero-downtime deployments and maintain service reliability during model updates. The DeepSeek-V3.1 model introduces hybrid thinking and non-thinking modes, enabling configurable chat templates for fast responses or deep reasoning, with built-in agent support and optimized tool-calling workflows for production use. This model is available via serverless APIs and Dedicated Endpoints, leveraging Together AI's infrastructure tuned for large MoE models.

All Together AI trails
  1. Canary rollouts: upgrade models in production without downtimeThis post introduces and details the implementation of canary rollouts for LLM inference endpoints. It explains the mechanics of staged traffic migration, including scaling up the target deployment before shifting traffic, health checks, propagation waits, and metric gates (e.g., p95 latency regression). It also contrasts canary rollouts with hard swaps and DIY staged cutovers, and provides examples of API, CLI, and Python SDK usage for creating, controlling, and managing rollouts, including pause, resume, promote, and cancel operations. The post also details the anatomy of a canary step and the differences between canary, blue-green, and rolling strategies.
  2. Autoscaling endpoints for LLM inferenceThis post elaborates on the autoscaling capabilities for LLM inference endpoints, detailing the trade-offs between over- and under-provisioning. It explains how to choose appropriate scaling metrics (concurrency-driven, SLO-driven, efficiency-driven) and tune scaling windows (`scale_up_window`, `scale_down_window`) to balance cost and latency. It also covers idle shutdown behavior, cold start phases, and the interaction of autoscaling with rollouts and traffic splitting.
  3. Configuring Dedicated Model InferenceIntroduced the architecture for dedicated model inference, comprising endpoints, deployments, and configs. Detailed the capacity-aware traffic split mechanism, where traffic is routed proportionally to a deployment's effective capacity (weight * ready_replicas). Explained how this enables seamless autoscaling composition, controlled per-replica load, and robust handling of non-ready replicas. Clarified the distinctions between traffic splits, A/B experiment percents, and canary step percentages. Provided an experiment demonstrating capacity-aware routing and a comparison of latency vs. throughput profiles for different configurations.
  4. ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at ScaleIntroduces ThunderAgent, a system that enhances agentic inference by treating agent workflows as schedulable programs rather than individual requests. This program-level abstraction addresses KV cache thrashing by selectively pausing low-priority workflows under memory pressure and routing resumed workflows to nodes with available capacity. ThunderAgent improves single-node throughput by up to 2.5x and achieves near-linear scaling on multi-node clusters, offering a drop-in solution compatible with existing inference engines and optimizations.
  5. Benchmarking inference at scale: coding agentsThis post introduces a new benchmark for coding agent workloads, emphasizing production-scale inference challenges like high concurrency, long context, and TTFT sensitivity. It details the engineering optimizations made to the Together Inference Engine, including ThunderMLA (a fused megakernel for MLA architecture) and custom kernel rewrites, which resulted in 31% more TPS and 2x better TTFT at saturation compared to TensorRT-LLM on the same hardware. The benchmark methodology and results are presented, highlighting the degradation curves of different engines under load. It also touches on cost savings and quality comparisons with other models.
  7. Foundational research powering efficient inference at scaleThis post details the critical role of inference optimization in AI product economics, highlighting its impact on latency, throughput, model landscape evolution, and concurrency. It emphasizes Together AI's compounding stack of research and systems engineering, including contributions to FlashAttention-4, ThunderKittens, and Aurora (adaptive speculative decoding). It also discusses full-stack hardware optimization on NVIDIA Blackwell, intelligent scheduling and batching, and the economic advantages of efficient inference, positioning Together AI's platform as an 'AI Native Cloud' for AI-native companies.
  8. Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM servingIntroduced Cache-Aware Prefill-Decode Disaggregation (CPD) to improve long-context LLM serving. CPD separates cold and warm workloads by cache hit rate, utilizing a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache via RDMA) and a router that steers requests to dedicated pre-prefill, prefill, or decode nodes. This architecture achieves up to 40% higher sustainable throughput and lower TTFT by preventing long-running cold prefills from blocking cache-reusing warm requests.
  9. Optimizing inference speed and costs: Lessons learned from large-scale deploymentsThis post details practical strategies for reducing inference latency and costs in large-scale LLM deployments. Key contributions include: 1. Model-level optimizations: Quantization (FP8/FP4) for 20-40% throughput improvement and distillation for creating lightweight, cost-effective variants (e.g., DeepSeek-R1). 2. Network latency reduction: Deploying regional inference proxies to shave 50-100ms off TTFT. 3. Reducing unnecessary compute: Addressing kernel inefficiencies, MoE memory stalls, prefill path issues, and batching/scheduling gaps. 4. Decoding optimizations: Utilizing MTP for multi-token prediction and speculative decoding with adaptive strategies (e.g., ATLAS) for 20-50% faster decoding. 5. Hardware and parallelism: Leveraging Blackwell GPUs and GB200 systems with tensor and expert parallelism for cost-per-token reduction. 6. Dynamic GPU capacity shifting: Reassigning GPUs across endpoints based on real-time demand to improve utilization and handle spikes.
  10. Introducing AutoJudge: Streamlined inference acceleration via automated dataset curationIntroduces AutoJudge, a method to accelerate LLM inference via task-specific lossy speculative decoding. AutoJudge automates dataset curation for speculative decoding by training a classifier to identify "important" mismatches that affect downstream quality, removing the need for manual annotation. It achieves 1.5-2x speedups over standard speculative decoding by accepting more draft tokens per verification cycle with minimal accuracy drop. The method integrates with existing frameworks like vLLM and TensorRT-LLM and can be composed with other techniques like EAGLE-2.
  11. Together AI delivers fastest inference for the top open-source modelsThis post details the engineering behind Together AI's achievement of up to 2x faster serverless inference for open-source LLMs, ranking #1 in output speed benchmarks. Key technical contributions include: 1. Next-gen GPU hardware optimization for NVIDIA Blackwell architecture (GB200 NVL72), including optimized paths for low-precision compute (FP8, FP4) and high-bandwidth data movement. 2. Development of 'Together Kernels' for NVIDIA Blackwell, including optimized FlashAttention-4 and fused MoE kernels. 3. A Turbo Optimization Suite featuring near-lossless quantization (FP8, FP4, hybrid precision) with architecture-aware calibration and fused FP4/FP8 kernels. 4. Production-grade speculative decoding algorithms with training-efficient methods, high-accuracy draft models, adaptive acceptance strategies, and fail-safe fallback mechanisms. 5. A scalable draft-model training pipeline supporting models up to 1T+ parameters, utilizing curriculum-based training, post-training recipes, data-mixing, and an alignment evaluation framework.
  12. DeepSeek-V3.1: Hybrid Thinking Model Now Available on Together AIThis post introduces DeepSeek-V3.1, a hybrid LLM with configurable chat templates for 'non-thinking' (fast response) and 'thinking' (deep reasoning) modes. It highlights built-in code and search agent capabilities with optimized tool-calling workflows. Performance benchmarks show significant improvements in 'thinking' mode for complex tasks like coding and reasoning. The model is deployed on Together AI's infrastructure with a 128K context window and MIT license, available via serverless APIs and Dedicated Endpoints.