Dedicated Model Inference Configuration

2 posts · Dec 2025 – Sep 2026

What this trail covers

Dedicated Model Inference (DMI) now offers full endpoint lifecycle control via API, UI, or CLI, enabling self-service scaling, custom-weight rollouts, and blue/green testing for customers. This includes programmatic access to usage and performance metrics, allowing teams to diagnose and resolve issues like queuing problems independently. DMI supports a rich library of frontier open-source models with performance-aware configurations, facilitating seamless transitions between model versions and context lengths. The platform is being adopted for diverse workloads, including coding assistants and customer-support NLP.

All Together AI trails

Sep 2026How a global fintech scaled coding agent traffic with Dedicated Model InferenceThis post details how a global fintech scaled their GLM 5.2 coding agent traffic using Together AI's Dedicated Model Inference (DMI). Key contributions include: 1. Self-service endpoint provisioning and scaling, allowing engineering teams to manage capacity without platform team intervention. 2. Programmatic access to metrics via a Metrics API for root cause analysis of issues like prefill backlogs and queuing. 3. Model fluidity demonstrated by seamless transitions from GLM 5.1 to GLM 5.2 and context length adjustments on the same endpoint. 4. The customer's deliberate choice to prioritize concurrency over raw throughput for their peak-load, low-TPS workload by selecting 256K/512K context over 1M. 5. The successful migration of a customer-support NLP workload to the same DMI infrastructure in a new region, showcasing the platform's versatility.
Jul 2026Configuring Dedicated Model InferenceThis post details the architecture of dedicated model inference on Together AI, introducing the concepts of endpoints, deployments, and configs. It explains the capacity-aware traffic splitting mechanism, where routing is proportional to `weight × ready_replicas`, enabling dynamic scaling and load balancing. The post also covers the selection of immutable configuration recipes (configs) based on optimization profiles (latency, throughput, balanced) and hardware choices, and demonstrates how to manage deployments and traffic splits using the CLI. It highlights the distinction between traffic split weights, A/B percentages, and canary steps, and provides an experiment showcasing the capacity-aware routing in action.
Jul 2026The production platform for open-weight AI inferenceThis post details significant enhancements to the inference platform, focusing on production-grade deployment and iteration. Key contributions include: advanced rollout strategies (canary, blue-green, rolling updates with auto-rollback based on thresholds), A/B and shadow testing against real traffic for evaluation, and more flexible autoscaling based on inference-native metrics (inflight requests, GPU utilization, time to first token, latency, decoding speed, throughput). It also highlights faster model startup times (4x improvement) due to rebuilt model caching and distribution, and the integration of custom training checkpoints directly into inference deployments, enabling a seamless transition from training to production.
Apr 2026Together AI Brings NVIDIA Nemotron 3 Nano Omni to Developers on Day 0This post announces the availability of NVIDIA Nemotron 3 Nano Omni on the Together AI platform, highlighting its multimodal reasoning capabilities across video, images, audio, and language. It emphasizes Together AI's research optimizations, including support for the model's hybrid Mamba-Transformer MoE architecture and multi-token prediction (MTP), which enable high throughput and cost-efficient inference. The platform's managed infrastructure is presented as ideal for agentic, production-scale inference workloads, offering predictable performance, high uptime, and token streaming. The post details how Nemotron 3 Nano Omni's unified context processing reduces system complexity and improves efficiency for multimodal agent applications.
Mar 2026Together AI Brings NVIDIA Nemotron 3 to Developers on Day 0This post announces the availability of NVIDIA Nemotron 3 Super on Together AI's Dedicated Inference platform. It highlights the model's hybrid Transformer+Mamba architecture, 1M-token context window, and multi-token prediction capabilities, emphasizing its suitability for agentic workflows and complex reasoning. The post details how Together AI's infrastructure, including the Together Inference Engine and custom CUDA kernels, optimizes the serving of this large model on single GPUs (H200/H100), providing production-grade isolation and a 99.9% uptime SLA.
Advertisement
Dec 2025Announcing native availability of NVIDIA Nemotron 3 Nano, NVIDIA’s latest reasoning modelThis post announces the native availability of NVIDIA Nemotron 3 Nano on Together AI. It details the model's hybrid Mamba-Transformer + sparse MoE architecture (~3B active parameters out of 30B) which enables fast, high-quality reasoning with a 1M-token context window. The post highlights Together AI's optimization for high throughput and cost-efficiency, making it ideal for agentic systems, coding assistants, and enterprise applications. It emphasizes performance, reliability, cost-efficiency, and flexible APIs for Nemotron 3 Nano deployment on the platform.