Dedicated Container Inference

5 posts · Feb 2026 – Aug 2026

What this trail covers

Dedicated Container Inference allows teams to deploy custom generative media models (video generation, avatar synthesis, image processing) with production-grade orchestration. It handles autoscaling, queuing, traffic isolation, and monitoring. Models trained on Together's GPU Cloud can be deployed directly without artifact transfers. The platform supports job orchestration for containers, enabling multiple independent queues, policy-driven traffic control, isolation between traffic types, and predictable behavior during spikes. It also offers volume mounts for model weights and built-in observability.

All Together AI trails

Aug 2026A/B test models in productionIntroduces A/B testing for LLM inference endpoints, allowing users to split live traffic into control and variant cohorts to measure model performance with real users. Details the mechanism of traffic subdivision, the requirement for variants to have zero weight in the base split, and the concept of fixed traffic shares independent of replica counts. Explains the process of ramping traffic, measuring results by joining platform metrics with product metrics, and ending experiments via promotion and deletion. Discusses edge cases like variant degradation, observed share accuracy, composition with rollouts, cohort stickiness, and autoscaling behavior. Demonstrates a full lifecycle example with configured vs. observed traffic shares at different stages.
Jul 2026Configuring Dedicated Model InferenceThis post details the architecture of Together AI's dedicated model inference platform, introducing the concepts of endpoints, deployments, and configs. It explains capacity-aware traffic splitting, where routing is proportional to a deployment's capacity (weight × ready replicas), enabling seamless autoscaling and load balancing. It also covers choosing between different config profiles (latency, throughput, balanced) and provides a practical example of capacity-aware routing in action.
Jul 2026The production platform for open-weight AI inferenceThis post details significant updates to Together AI's inference platform, focusing on production-grade deployment features for open-weight and licensed models. New capabilities include canary, blue-green, and rolling updates with auto-rollback, A/B and shadow testing against real traffic, and advanced autoscaling based on various inference-native metrics. It also introduces a closed beta for custom training (full-weight and LoRA RL, supervised fine-tuning) with direct deployment of checkpoints. The platform now offers faster model startup times (4x improvement) due to rebuilt model caching and distribution, and exposes a Prometheus endpoint for organization-level observability.
May 2026Deploy and inference any model from HuggingFaceThis post demonstrates an agentic workflow using Goose and the Together dedicated containers skill to deploy arbitrary HuggingFace models to Together's Dedicated Container Inference (DCI) infrastructure. It highlights how an agent can automate the process of configuring inference servers, container specifications, and wiring up deployments for new models, reducing the time from model release to production readiness from days to minutes. The post also provides an example of submitting an asynchronous inference job for a video generation model and retrieving the output.
Mar 2026Together AI Brings NVIDIA Nemotron 3 to Developers on Day 0This post announces the availability of NVIDIA Nemotron 3 Super on Together AI's Dedicated Inference platform. It details the model's hybrid MoE architecture (Transformer + Mamba), 1M-token context window, and multi-token prediction capabilities, highlighting their benefits for agentic workflows and complex reasoning. The post also explains how Nemotron 3 Super is optimized for single-GPU deployment on H200/H100 GPUs within Together AI's managed infrastructure, emphasizing the use of the Together Inference Engine and custom CUDA kernels for accelerated performance and production-grade isolation with an SLA and SOC 2 compliance.
Advertisement
Feb 2026Introducing Dedicated Container Inference: Delivering 2.6x faster inference for custom AI modelsThis post introduces Dedicated Container Inference (DCI) as a new capability for deploying custom generative media models. DCI is built around job orchestration for containers, enabling multiple independent queues, policy-driven traffic control, isolation between batch, real-time, and untrusted traffic, and predictable behavior during spikes. It supports deploying Docker containers with inference logic, managing deployment, autoscaling, queuing, and monitoring at the job level. Key architectural components include volume mounts for model weights, inference running as jobs, autoscaling driven by queue depth or custom metrics, explicit traffic policies, and built-in observability. The post highlights performance gains seen by customers, with up to 2.6x inference speedups for video generation models.