Together AI Feature Trails

The Together Fine-Tuning Platform now supports tool call, reasoning, and vision-language model (VLM) fine-tuning. This expansion enables more reliable agentic workflows by ensuring agents execute structured actions reliably with end-to-end fine-tuning and inference on OpenAI-compatible schema, specialized support for training models on 'thinking' tokens in reasoning traces, and native support for vision training to align vision-language models with complex, domain-specific visual data. This post also details a cost-performance analysis of different LLMs for coding tasks, demonstrating a cascade strategy using DeepSeek-V4 Flash and GPT-5.6 Luna to achieve superior accuracy and cost-efficiency.

Together AI now offers native deployment of elite proprietary models like Kimi K3, featuring advanced coding capabilities and competitive performance against closed models like Claude Fable 5. This includes optimized inference for open-weight models, enabling cost-effective, high-volume agentic workflows. Kimi K3 demonstrates strong performance in software engineering tasks, particularly with multiple attempts (pass@2, pass@4), and offers significant cost savings compared to proprietary alternatives. The platform supports serving open models at production scale, allowing users to leverage their capabilities for tasks requiring retry tolerance or broad coverage.

Autoscaling for LLM inference endpoints on Together AI allows users to configure deployments to automatically scale based on inference-native metrics such as in-flight requests, TTFT, GPU utilization, and token throughput. Users can set replica bounds, choose a scaling metric and target, and tune scale-up and scale-down windows to manage traffic fluctuations and optimize for latency and cost. The system supports various scaling metrics including concurrency-driven, SLO-driven, and efficiency-driven. This post details a benchmark comparing Together Inference Engine against TensorRT-LLM and SGLang on a production coding agent workload, demonstrating significant improvements in TPS and TTFT under high concurrency and long context scenarios. It highlights full-stack optimization techniques like ThunderMLA and custom kernel rewrites, and discusses the importance of production-relevant benchmarks that consider factors like KV cache pressure and prefill-heavy output shapes. The post also touches on the cost-effectiveness of their models compared to proprietary alternatives.

Together AI has significantly updated its inference platform, offering enhanced control over performance, cost, and quality for open-weight and licensed models. Key improvements include production-grade deployment features like canary, blue-green, and rolling updates with auto-rollback, A/B and shadow testing for real traffic evaluation, and advanced autoscaling capabilities. The platform now supports custom training betas for full-weight and LoRA reinforcement learning and supervised fine-tuning, with seamless deployment of trained checkpoints. Model caching and distribution have been rebuilt for faster warm starts (up to 4x). The platform provides an organization-level Prometheus endpoint for observability and an improved in-product analytics experience. This update aims to simplify the transition from experimentation to production, enabling continuous model improvement without requiring users to build their own inference stack.

Together GPU Clusters (formerly Instant Clusters) now integrate autoscaling powered by Kubernetes Cluster Autoscaler, Role-Based Access Control (RBAC) for structured multi-team governance, full-stack observability with dedicated Grafana instances, and self-serve node repair with active health checks and acceptance testing to reduce Mean Time To Repair (MTTR) for hardware failures. These enhancements aim to provide the elasticity of a virtualized stack with the performance of bare metal, enabling production-ready managed infrastructure for large-scale distributed training and variable inference workloads.

Together AI now offers a detailed breakdown of its inference reliability tiers, explaining the architectural requirements for achieving 99%, 99.9%, and 99.99% uptime. The company emphasizes its ownership of the full stack, from chip to token, and its commitment to transparently defining what each SLA tier covers, including infrastructure ownership, failover testing, and measurement at inference completion. This ensures customers understand the concrete engineering behind their uptime guarantees.

Together AI now offers native deployment of elite proprietary image and video generation models, including OpenAI's Sora 2 Pro, Google's Veo 3.0 and Imagen 4.0 Ultra, and ByteDance's Seedream 1.0 Pro. This enables high-quality, production-grade media generation with advanced features like multi-shot storytelling, cinematic quality, and advanced camera control. The platform provides unified tooling, SDKs, authentication, and billing for multimodal workloads, allowing for tighter creative control. Additionally, Together AI now offers native deployment of Thinking Machines Lab's Inkling model, a multimodal mixture-of-experts model supporting text, image, and audio inputs with controllable inference effort and a 1M context window. Inkling features query-conditioned relative attention, short causal convolutions, and a shared-sink MoE architecture, optimized for efficient production inference on Together AI's platform using a FlashAttention-4-based kernel.

Together AI now offers Provisioned Throughput, a new inference form factor for frontier open models. This provides guaranteed capacity with a 99% uptime SLA and token-based pricing, bridging the gap between best-effort serverless and complex dedicated inference. It offers predictable costs, up to 90% lower than proprietary alternatives, and is available for models like MiniMax M3 and GLM-5.2. Provisioned Throughput Units (PTUs) are consumed based on input, cached input, and output tokens, allowing for efficient serving of models with advanced features like 1M context windows and multimodality. This includes deep performance optimizations such as KV-Block-Major sparse attention kernels, paged attention integration, optimized index scoring kernels, and Rust-based multimodal preprocessing gateways, resulting in significant throughput improvements.

FlashAttention-4 introduces algorithm and kernel co-design for asymmetric hardware scaling, specifically targeting Blackwell GPUs. It features new pipelining for maximum overlap of tensor cores, softmax exponential, and memory operations. The forward pass includes software emulation of the exponential function and conditional online softmax rescaling. The backward pass utilizes tensor memory to store intermediate results, reducing shared-memory traffic, and supports a 2-CTA MMA mode. New tile sc. This post details the development and impact of the Kernels Lab, highlighting their work on FlashAttention and the ThunderKittens library for optimizing NVIDIA Blackwell GPUs. It also showcases the Together Megakernel implementation for real-time voice agent workloads, achieving significant latency reductions. The team's approach emphasizes academic-industry symbiosis and customer-facing collaboration for custom kernel optimization.

This post introduces the concept of hardware utilization and software-hardware co-design as a key driver for future AI progress, challenging the notion of hitting a hardware wall. It highlights that current AI systems are significantly underutilizing existing hardware (e.g., ~20% MFU for training, single digits for inference) and that future compute generations are yet to be fully leveraged. The post also points to the utility of current models in complex tasks like GPU kernel generation, suggesting a new benchmark, ParallelKernelBench (PKB), for evaluating LLM capabilities in generating multi-GPU kernels. PKB reveals that current frontier LLMs struggle with multi-GPU kernel generation, often failing to compile or produce incorrect results, and rarely outperform naive baselines. This highlights a gap in LLM reasoning about rank coordination, data partitioning, and collective ordering, as well as a lack of proficiency with specialized communication mechanisms like TMA and NVLS.

Together AI now offers native deployment of elite proprietary Speech-to-Text (STT) and Text-to-Speech (TTS) models, including Whisper (streaming), Orpheus (high-fidelity TTS), Kokoro (ultra-low latency TTS), and Voxtral Mini (multilingual transcription with speaker diarization), on dedicated infrastructure. This enables high-quality, low-latency voice generation and transcription with advanced features like WebSocket streaming for real-time audio processing, optimized voice activity detection, and now, a comprehensive video translation pipeline (Violin) leveraging these core capabilities. Violin integrates ASR (Whisper V3), LLM translation (Deepseek V4 Pro), and TTS (Cartesia's Sonic 3) to break language barriers in video content, offering features like video-content-aware chat assistants powered by vision-language models (Qwen3.5-397B-A17B) and voice style personalization. The Violin pipeline is available as a web app, CLI tool, and agent skill, all open-source.

Together AI now offers native deployment of elite proprietary Text-to-Speech (TTS) models, including Rime's Arcana V3 Turbo (English-Spanish code-switching, ~120ms TTFA) and Arcana V3 (11-language switching, ~160ms TTFA), on dedicated infrastructure. These models feature natural code-switching that maintains cadence and prosody across language boundaries, enabling more conversational and trustworthy voice agents. They are co-located with LLM and STT workloads for unified API, authentication, and billing. A new 'Voice Finder' tool has been introduced to simplify the selection of over 600+ voices across multiple TTS models by allowing search via prompt or audio sample, and by providing model-aware metadata across 15+ dimensions (pitch, accent, language, age, emotion, speaking style). This tool aims to accelerate the process of finding the right voice for specific use cases, improving the overall voice agent development experience.

Together AI now offers native deployment of elite proprietary models like DeepSeek V4 Pro, featuring a 512K-token context window (model-level 1M) for long-context reasoning. This model utilizes a large-scale MoE architecture (1.6T parameters, 49B activated) with hybrid attention (Compressed Sparse Attention and Heavily Compressed Attention) to manage serving costs, memory pressure, KV cache usage, and latency. It supports controllable reasoning modes (Non-Think, Think High, Think Max) to match reasoning depth to task difficulty. Pricing includes transparent serverless options with a 90% cost reduction for reused context via cached input pricing. The platform supports a deployment path from Serverless Inference to Dedicated Infrastructure for reserved capacity and production control, catering to workloads like code agents, document intelligence, and research synthesis.

Together AI has partnered with Cursor to build and deploy real-time, low-latency inference infrastructure on NVIDIA Blackwell GB200 NVL72 and HGX B200. This includes full throughput on ARM hosts, custom kernels for Blackwell Tensor Cores, and efficient parallelism across GPUs. The platform also supports a streamlined weights-to-production cycle with quantization pipelines using NVIDIA TensorRT LLM, enabling rapid iteration and A/B testing of new models. The focus is shifting towards increasing throughput and reducing latency through techniques like quantization, distillation, regional inference proxies, kernel optimization, advanced decoding strategies (MTP, speculative decoding), strategic hardware selection (Blackwell, GB200), appropriate parallelism (tensor, expert), and dynamic GPU capacity shifting across endpoints. This post details lessons learned from large-scale deployments on optimizing inference speed and costs by maximizing GPU utilization, eliminating compute stalls, strategically selecting decoding techniques, and treating performance tuning as an ongoing discipline.

Together AI has implemented a robust strategy for managing kernel-level security vulnerabilities, exemplified by their rapid response to the Copy Fail (CVE‑2026‑31431) vulnerability. This includes immediate fleet-wide mitigation by disabling vulnerable kernel modules, followed by a phased rollout of vendor-provided kernel patches. The platform emphasizes a default-off approach for niche kernel interfaces and maintains continuous monitoring for anomalous kernel behavior and usage. This proactive stance ensures the security and stability of multi-tenant AI infrastructure.

Together AI is exploring and enabling advanced model architectures like Parcae, which introduce stable looped language models. This approach allows for achieving higher quality with fewer parameters by increasing recurrence rather than solely scaling data, opening efficient frontiers for training memory-constrained on-device models. The research focuses on stabilizing training for these looped architectures, establishing new scaling laws, and demonstrating their effectiveness in matching or exceeding performance of larger models. This partnership with Meta's PyTorch team will integrate sophisticated high-performance Reinforcement Learning (RL) into the Together AI platform, complementing existing services like supervised fine-tuning, sandboxed environments, evaluation tools, model optimizations, and high-performance inference. The goal is to provide a complete platform for building, training, and deploying next-generation agentic systems with advanced RL capabilities.

Together AI enables the creation of autonomous data scientist agents by integrating open-source LLMs with a secure code execution environment (Together Code Interpreter - TCI). These agents leverage the ReAct (Reasoning and Action) pattern, specifically the CodeAct variant, to perform multi-step data analysis tasks. The system allows for flexible prompt engineering to tune agent behavior and supports dynamic installation of necessary libraries within the sandboxed execution environment. This post introduces DSGym, a unified framework that integrates diverse data science evaluation suites behind a single API with standardized abstractions for datasets, agents, and metrics. DSGym unifies and refines existing benchmarks while expanding the scope with novel scientific analysis tasks (90 bioinformatics tasks from academic literature) and challenging end-to-end modeling competitions (92 Kaggle competitions). Beyond evaluation, DSGym provides trajectory generation and synthetic query pipelines for agent training—we demonstrate this by training a 4B model on 2k generated examples, achieving state-of-the-art performance among open-source models. The framework abstracts code execution behind containers, comes with pre-installed dependencies, and offers a unified JSON interface for all benchmarks. DSGym categorizes tasks into Data Analysis and Data Prediction, and introduces DSBio and DSPredict datasets. It also includes a data generation pipeline for agent training, producing 2,000 high-quality query-trajectory pairs for supervised finetuning. The post addresses the 'memorization gap' by filtering out tasks solvable without data access and presents benchmark performance and failure modes, showing that even frontier models underperform on multi-step reasoning, with scientific analysis tasks dominated by domain-grounding errors. DSPredict evaluates the ability of agents to build complete machine learning pipelines, with performance measured by valid submission, median/percentile, and medal achievements.

Together AI is exploring the application of LLMs to optimize database query execution plans. This involves developing harnesses like DBPlanBench to expose query execution processes to LLMs, enabling them to analyze and suggest rewrites to improve performance. The approach focuses on semantic reasoning over statistical estimation, using techniques like JSON patching for targeted edits and evolutionary search for iterative refinement. The goal is to enhance query performance and reduce resource consumption without modifying the core database engine.

Together AI is exploring and implementing advanced techniques to accelerate LLM inference. This includes methods like AutoJudge, which automates dataset curation for lossy speculative decoding, enabling significant speedups (1.5-2x) by identifying and accepting task-specific, non-critical token mismatches. This approach integrates with existing frameworks like vLLM and TensorRT-LLM, and can be combined with other acceleration techniques like EAGLE-2. Together AI now delivers up to 2x faster serving. This post introduces ATLAS (AdapTive-LeArning Speculator System), a novel speculative decoding system that dynamically improves at runtime by learning from historical patterns and live traffic, offering up to 2.65x speedup on models like DeepSeek-V3.1 and Kimi-K2, outperforming even specialized hardware.

Together AI introduces Consistency Diffusion Language Models (CDLM), a post-training recipe that significantly accelerates inference for Diffusion Language Models (DLMs). CDLM combines consistency-based multi-token finalization with block-wise KV caching, achieving up to 14.5x latency speedups on math and coding tasks. This method addresses the inefficiencies of standard DLMs by enabling exact block-wise KV caching and reducing the high number of refinement steps required for quality generation. CDLM trains a block-wise causal student model by minimizing distillation loss for newly unmasked positions, consistency loss for still-masked positions, and an auxiliary DLM masked-denoising loss. At inference, CDLM decodes in a block-wise autoregressive manner with KV caching for finalized blocks and parallel finalization within each block. System-level analysis shows that block-wise DLMs occupy a balanced operating point for small-batch settings, offering improved arithmetic intensity over autoregressive models while remaining efficient.

Together Evaluations now supports OpenAI, Anthropic, and Google models for comprehensive benchmarking. This enables users to compare any models side-by-side—open-source, fine-tuned, or proprietary—to make data-driven decisions on quality, cost, and performance. The framework now supports evaluating Together fine-tuned models and includes new recipes for optimizing and evaluating open-source models, demonstrating how fine-tuned open models can outperform proprietary judges at significantly lower cost and higher speed, and how automated prompt optimization can deliver substantial quality gains.

Together AI has released Python SDK v2.0, a new, type-safe, OpenAPI-generated client. This version offers improved architecture, better type safety and developer experience, future-proofing for new features, and compatibility with modern tools like `uv`. It utilizes `httpx` for better performance and includes enhanced error handling, a modern parameter system with keyword-only arguments, and new capabilities like raw response access, streaming helpers, and code interpreter session management. Many existing APIs are drop-in compatible, while others have updated signatures. New beta APIs for Instant Clusters are also included.

Together AI now offers native deployment of elite proprietary Text-to-Speech (TTS) models, including MiniMax Speech 2.6 Turbo, on dedicated infrastructure. This enables high-quality, low-latency voice generation with advanced features like multilingual streaming, emotional awareness, and rapid voice cloning, integrated alongside LLM and STT workloads for a unified real-time voice agent pipeline. The Batch Inference API has been significantly improved with a streamlined UI for easier job creation and tracking. It now supports all serverless models and private deployments, offering universal model access. Rate limits have been increased by 3000x, from 10M to 30B enqueued tokens per model per user. The API also offers lower costs, typically 50% of the real-time API for most serverless models, making it an economical choice for high-throughput workloads. Ideal use cases include large-scale batch inference on models like DeepSeek-R1-0528, leveraging NVIDIA Blackwell GPUs and optimized inference engines for top speeds.

Together AI now offers VirtueGuard, an enterprise-grade AI security and safety model, integrated directly into its platform. This guardrail model provides comprehensive protection against harmful outputs, compliance violations, and reputational damage. It boasts an 8ms response time, significantly faster than alternatives, with a high F1 score and low false positive rate. VirtueGuard covers 12 risk categories across text, images, and audio. The integration is seamless, requiring only a single API parameter, and benefits from Together AI's 99.9% uptime SLA, automatic scaling, and continuous improvement from Virtue AI's research team. It is proven at scale with organizations like Uber, Anthropic, NVIDIA, and Glean.

Together AI introduces a Batch API for processing large volumes of LLM requests asynchronously. This feature allows users to submit up to 50,000 requests in a single batch file (up to 100MB) for non-urgent workloads, offering a 50% cost reduction compared to real-time inference. The API supports progress tracking through various stages (validation, queuing, processing, aggregation) and provides results within a best-effort 24-hour window. It integrates with the Files API for uploads and job creation, and the `together` Python client for programmatic interaction. The Batch API has dedicated rate limits independent of real-time usage, including a maximum of 10 million tokens enqueued per model.