Together AI Feature Trails

Feature trailActive

Instant GPU Clusters

Together GPU Clusters now integrate autoscaling, Role-Based Access Control (RBAC), full-stack observability, self-serve node repair, and active health checks. Autoscaling leverages Kubernetes Cluster Autoscaler for elastic capacity. Health checks and self-repair reduce Mean Time To Repair (MTTR) for hardware failures. RBAC provides structured multi-team governance through projects and roles (Admin, Member). Full-stack observability is provided via dedicated Grafana instances with pre-built dashb. This collaboration expands enterprise-grade inference capacity by deploying large clusters of NVIDIA B300 GPUs on IBM Cloud, leveraging NVIDIA Spectrum-X Ethernet networking for high-throughput inference.

12 posts · 2025–2026

Timeline

20252026

Feature trailActive

Agentic Tool Integration

Together Link enables engineering teams to integrate open-source LLMs into their existing coding agent workflows, reducing costs by over 50% and providing access to frontier models like Kimi K3 and GLM 5.3 for complex tasks, while smaller models handle everyday coding. The system supports popular agents like Claude Code, Codex, and OpenCode, routing tasks to the most appropriate model based on session context. Billing is managed through existing Together AI API keys.

1 post · 2026

Timeline

Feature trailActive

Fine-Tuning Platform

The Together Fine-Tuning Platform now supports advanced fine-tuning workflows for open-source LLMs, enabling them to outperform larger proprietary models on specialized tasks. This includes rigorous evaluation-driven SFT and RL optimization, leading to significant improvements in accuracy (up to 60%), reduced inference costs (10-100x), and enhanced transparency and reliability. The platform facilitates seamless integration of domain-specific expertise with production-ready fine-tuning and inference. This post details the process of fine-tuning a Qwen3.5 4B model for classification tasks, demonstrating how to prepare datasets, launch training jobs, and deploy the resulting model to a dedicated endpoint, with a focus on cost-effectiveness ($17 training cost).

10 posts · 2025–2026

Timeline

20252026

Feature trailActive

Autoscaling for LLM Inference Endpoints

Together AI offers advanced deployment strategies for LLM inference endpoints, including canary rollouts with metric gates for safe model upgrades, blue-green deployments for rapid switching, and rolling updates for in-place replica swaps. These strategies allow for controlled traffic migration, automated health and performance checks, and robust rollback mechanisms to ensure zero-downtime deployments and maintain service reliability during model updates. The DeepSeek-V3.1 model introduces hybrid thinking and non-thinking modes, enabling configurable chat templates for fast responses or deep reasoning, with built-in agent support and optimized tool-calling workflows for production use. This model is available via serverless APIs and Dedicated Endpoints, leveraging Together AI's infrastructure tuned for large MoE models.

7 posts · 2025–2026

Timeline

20252026

Feature trailActive

Dedicated Model Inference Configuration

Dedicated Model Inference (DMI) now offers full endpoint lifecycle control via API, UI, or CLI, enabling self-service scaling, custom-weight rollouts, and blue/green testing for customers. This includes programmatic access to usage and performance metrics, allowing teams to diagnose and resolve issues like queuing problems independently. DMI supports a rich library of frontier open-source models with performance-aware configurations, facilitating seamless transitions between model versions and context lengths. The platform is being adopted for diverse workloads, including coding assistants and customer-support NLP.

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

Open Source AI Stack

The Together AI platform supports a wider range of open-weight models, including GLM-5.3 and GLM-5.3 Flash, with enhanced experiment tracking that provides live metrics for loss, gradient norm, and learning rate. New controls include expert LoRA for Mixture-of-Experts models, early stopping based on validation loss, and arbitrary batch size support via gradient accumulation. Data processing is more transparent with tokenized data inspection and the ability to download prepared datasets. This release also details a successful migration strategy from closed-source to open-source models, emphasizing a phased approach involving discovery, evaluation, adaptation, decision-making, and production deployment. The strategy highlights the importance of defining use-cases, leveraging benchmarks, performing custom evaluations with real data, adapting prompts and inference settings, and communicating ROI to stakeholders. Managed services are presented as a way to accelerate this migration.

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

Hardware Utilization and Software-Hardware Co-design

3 posts · 2025–2026

Timeline

20252026

Feature trailActive

LLM Model Performance and Cost Comparison

Together AI continues to provide in-depth comparisons of LLM performance and cost, now detailing the trade-offs between DeepSeek V4 Flash 0731 and GPT-5.6 Luna on the DeepSWE benchmark. The analysis highlights DeepSeek V4 Flash's strength in cost-effectiveness and cleaner failure modes, while GPT-5.6 Luna excels in accuracy and broader domain coverage. The post advocates for a cascaded routing strategy, using DeepSeek V4 Flash as a first stage to achieve superior accuracy at a lower cost than us. This post also compares OpenAI's new gpt-oss-120B model against o4-mini across practical tests, showing gpt-oss-120B's strengths in code generation and instruction following, while o4-mini excels in creative SVG generation. The results reinforce the value of open-source models for customization, cost-efficiency, and deployment flexibility.

4 posts · 2025–2026

Timeline

20252026

Feature trailActive

Long Context Task Decomposition

Together AI now offers native deployment of elite proprietary models like DeepSeek V4 Pro and MiniMax M3, featuring up to 1.05M-token context windows for long-context reasoning and advanced software engineering capabilities. This post details the engineering breakthroughs for serving MiniMax M3 efficiently, including a KV-Block-Major sparse attention kernel, paged attention integration for MSA, optimized index scoring, and a Rust-based multimodal preprocessing gateway, resulting in 81–125% throughput improvements. DeepSeek V4 Pro is now available with a 512K-token context window, with full 1M context available on dedicated infrastructure. The model utilizes a large-scale MoE architecture (1.6T parameters, 49B activated) and offers controllable reasoning modes (Non-Think, Think High, Think Max). Cached input pricing offers a significant cost reduction for repeated context usage.

6 posts · 2026

Timeline

Feature trailActive

A/B Testing for LLM Endpoints

Together AI now offers A/B testing for LLM endpoints, allowing users to split live traffic into cohorts to measure candidate model performance with real users. This feature enables controlled ramp-ups, easy promotion of winning models via blue-green rollouts, and clean deletion of experiments. It integrates with existing platform metrics for comprehensive analysis and provides sticky cohorts per user if a sampling key is provided. The platform also supports multi-way tests with up to 20 variants and ensures consistent traffic shares independent of replica counts. Updates are guarded by etags for concurrent ramp safety, and the system handles experiment deletion gracefully, returning traffic to the base split.

1 post · 2026

Timeline

Feature trailActive

Dedicated Container Inference

Dedicated Container Inference allows teams to deploy custom generative media models (video generation, avatar synthesis, image processing) with production-grade orchestration. It handles autoscaling, queuing, traffic isolation, and monitoring. Models trained on Together's GPU Cloud can be deployed directly without artifact transfers. The platform supports job orchestration for containers, enabling multiple independent queues, policy-driven traffic control, isolation between traffic types, and predictable behavior during spikes. It also offers volume mounts for model weights and built-in observability.

5 posts · 2026

Timeline

Feature trailActive

Kimi K3 Model Serving

Together AI now offers native deployment of elite proprietary models like Kimi K2, featuring a 1 trillion parameter scale and advanced agentic capabilities. This model competes at the GPT 5.6 Sol and Claude Fable 5 tier, excelling in EQ-Bench3, Creative Writing, SWE-bench Verified, AceBench, and LiveCodeBench v6. It utilizes a Mixture-of-Experts design with 8 experts selected from 384, trained on 15.5 trillion tokens using the MuonClip optimizer. The platform provides serverless deployment with 99.9% availability SLA, instant scaling, and continuous performance optimizations, offering competitive economics compared to closed models.

13 posts · 2025–2026

Timeline

20252026

Feature trailActive

Agentic Inference Scheduling

Together AI continues to advance agentic inference scheduling with the introduction of ThunderAgent, a novel program abstraction for agentic LLM request scheduling. This system significantly improves throughput and reduces latency for agentic workloads by treating agentic workflows as schedulable programs. It addresses KV cache thrashing by pausing low-priority workflows under memory pressure and resuming them onto nodes with available capacity, leading to near-linear scaling on multi-node clusters and compatibility with existing inference optimizations. This builds upon previous work in AI agent orchestration and autonomous data scientist agents.

1 post · 2025–2026

Timeline

20252026

Feature trailActive

Inference Reliability and Uptime

Together AI now offers a detailed breakdown of its inference reliability tiers, explaining the architectural requirements for achieving 99%, 99.9%, and 99.99% uptime. The company emphasizes its ownership of the full stack, from chip to token, and its commitment to transparently defining what each SLA tier covers, including infrastructure ownership, failover testing, and measurement at inference completion. This ensures customers understand the concrete engineering behind their uptime guarantees. This post specifically details the engineering behind achieving 99.9% uptime by surviving full data center failures through multi-facility deployment and continuous live traffic routing, and outlines the questions customers should ask providers regarding infrastructure ownership, full-stack expertise, capacity, and SLA measurement.

1 post · 2026

Timeline

Feature trailActive

Production GPU Cluster Reliability and Control

Together GPU Clusters now offer enhanced reliability and operational control for production workloads. This includes passive health checks and auto node repair to catch and recover from hardware failures, a rebuilt Slurm-on-Kubernetes stack (Slinky 1.0) for improved worker daemon resilience, durable job accounting, reliable process cleanup, and accurate GPU state management after reschedules. Operational control features include a new cluster details view with health and usage metrics, external OIDC for Kubernetes RBAC for granular access control, and startup scripts for cluster customization at boot, job start, and job end.

1 post · 2026

Timeline

Feature trailActive

Video Generation Models

Together AI now offers native deployment of elite proprietary image and video generation models, including FLUX.2 from Black Forest Labs, OpenAI's Sora 2 Pro, Google's Veo 3.0 and Imagen 4.0 Ultra, ByteDance's Seedream 1.0 Pro, and the Wan 2.7 suite. This expansion includes over 40 new models, enabling complete workflows that combine text, image, and video generation within a single application. The platform provides OpenAI-compatible endpoints, unified authentication, and transparent per-model usage. FLUX.2 specifically targets production-grade image generation with multi-reference input for character/product consistency, hex code color matching for brand compliance, and improved text rendering. It is available in Dev, Pro, and Flex variants, served through Together AI's fast, reliable infrastructure with the same APIs used across the rest of the generative stack.

6 posts · 2025–2026

Timeline

20252026

Feature trailActive

Provisioned Throughput for LLM Inference

Together AI now offers Provisioned Throughput for frontier open models, providing guaranteed capacity with a 99% uptime SLA and token-based pricing. This bridges the gap between serverless and dedicated inference, offering predictable costs and a simpler alternative to managing dedicated inference infrastructure. It is available for models like MiniMax M3 and GLM-5.2, with capacity across multiple regions and a one-month minimum term. The pricing model accounts for input, cached input, and output tokens, allowing for optimization based on traffic patterns.

2 posts · 2026

Timeline

Feature trailActive

Multi-GPU Kernel Generation Benchmark

Together AI is developing and evaluating benchmarks for multi-GPU kernel generation, such as ParallelKernelBench (PKB), to assess the capabilities of frontier LLMs in writing production-ready CUDA kernels for distributed AI workloads. This involves creating a diverse set of problems derived from real codebases and evaluating models on correctness, speedup, and communication efficiency across various parallelism strategies.

1 post · 2026

Timeline

Feature trailActive

Multi-Node Model Training

This post details the infrastructure, techniques, and practical steps for distributed training of foundation models across multi-node GPU clusters. It covers parallelism strategies (data, tensor, pipeline), network interconnects (NVLink, InfiniBand), checkpointing and fault tolerance, and practical considerations for scaling training from billions to trillions of parameters. It also includes a production example of training Qwen2.5-72B and common failure modes.

1 post · 2026

Timeline

Feature trailActive

Speech-to-Text (STT) APIs

Together AI now offers native deployment of elite proprietary Speech-to-Text (STT) and Text-to-Speech (TTS) models, including Deepgram's Nova-3, Nova-3 Multilingual, Flux, and Aura-2, on dedicated infrastructure. This enables high-quality, low-latency voice generation and transcription with advanced features like WebSocket streaming for real-time audio processing, optimized voice activity detection, and speaker diarization. The integration allows for co-location with LLM and TTS workloads for unified pipelines, offering production-grade reliability with a 99.9% uptime SLA, SOC 2 Type II and HIPAA-ready support, and global regions with data residency options. The platform provides a single production surface for real-time voice agents, with STT, LLM, and TTS on one platform, simplifying operations and control over performance.

8 posts · 2025–2026

Timeline

20252026

Feature trailActive

Real-time Low-Latency Inference Infrastructure

Together AI has partnered with Cursor to build and deploy real-time, low-latency inference infrastructure on NVIDIA Blackwell GB200 NVL72 and HGX B200. This includes full throughput on ARM hosts, custom kernels for Blackwell Tensor Cores, and efficient parallelism across GPUs. The platform also supports a streamlined weights-to-production cycle with quantization pipelines using NVIDIA TensorRT LLM, enabling rapid iteration and A/B testing of new models. The focus is shifting towards increasing throughput and reducing latency for LLM inference, leveraging bespoke GPU kernels, a proprietary inference engine, advanced speculative decoding methods, and calibrated/quantized model optimization.

3 posts · 2025–2026

Timeline

20252026

Feature trailActive

Kernel Security and Vulnerability Management

Together AI has implemented a robust strategy for managing kernel-level security vulnerabilities, exemplified by their rapid response to the Copy Fail (CVE‑2026‑31431) vulnerability. This includes immediate fleet-wide mitigation by disabling vulnerable kernel modules, followed by a phased rollout of vendor-provided kernel patches. The platform emphasizes a default-off approach for niche kernel interfaces and maintains continuous monitoring for anomalous kernel behavior and usage. This proactive approach ensures the security and stability of their AI infrastructure.

1 post · 2026

Timeline

Feature trailActive

Speculative Decoding Acceleration

Together AI introduces ATLAS (AdapTive-LeArning Speculator System), a novel runtime-learning accelerator for LLM inference. ATLAS dynamically improves speculative decoding performance by learning from historical patterns and live traffic, continuously aligning with the target model's behavior. This adaptive approach offers significant speedups compared to standard or custom speculators, outperforming specialized hardware. The system integrates with existing Together Turbo techniques and aims to reduce inference latency and costs.

5 posts · 2025–2026

Timeline

20252026

Feature trailActive

Looped Language Models

Together AI continues to explore and enable advanced model architectures like Parcae, which introduce stable looped language models. This approach allows for achieving higher quality with fewer parameters by increasing recurrence rather than solely scaling data, opening efficient frontiers for training memory-constrained on-device models. The research focuses on stabilizing training for these looped architectures, establishing new scaling laws, and demonstrating their effectiveness in matching or exceeding the performance of larger fixed-depth models. Together AI is releasing training code and models for Parcae to accelerate research in parameter efficiency.

3 posts · 2025–2026

Timeline

20252026

Feature trailActive

Autonomous Data Scientist Agent

Together AI enables the creation of autonomous data scientist agents by integrating open-source LLMs with a secure code execution environment (Together Code Interpreter - TCI). These agents leverage the ReAct (Reasoning and Action) pattern, specifically the CodeAct variant, to perform multi-step data analysis tasks. The system allows for flexible prompt engineering to tune agent behavior and supports dynamic installation of necessary libraries within the sandboxed execution environment. DSGym is used for evaluation.

6 posts · 2025–2026

Timeline

20252026

Feature trailActive

AI Native Cloud

The AI Native Cloud is a purpose-built cloud infrastructure designed for AI-native companies. It addresses the unique challenges of the AI lifecycle, including rapid iteration, continuous research integration, massive scale, and developer velocity. Key characteristics include a full AI stack from hardware to software, a fast path from research to production, reliability at massive scale, a builder-centric design, and a partnership model that moves at the pace of AI-native growth.

1 post · 2026

Timeline

Feature trailActive

LLM-Guided Database Query Optimization

Together AI is exploring the application of LLMs to optimize database query execution plans. This involves developing harnesses like DBPlanBench to expose query execution processes to LLMs, enabling them to analyze and suggest rewrites to improve performance. The approach focuses on semantic reasoning over statistical estimation, using techniques like JSON patching for targeted edits and evolutionary search for iterative refinement. The goal is to enhance query performance and reduce resource consumption, with a practical workflow for transferring optimizations from smaller scale factors to larger ones.

1 post · 2026

Timeline

Feature trailActive

FlashAttention-4 Algorithm and Kernel Co-Design

Together AI continues to push the boundaries of AI hardware utilization and software-hardware co-design, exemplified by their work on the NVIDIA Vera Rubin NVL72 platform. This post details the integration of ThunderKittens, a GEMM kernel, with the new ISA and features of Vera Rubin, achieving over 22 PFLOPS in NVFP4 GEMMs. The team has optimized the kernel by leveraging doubled K-step instructions, increased tensor memory, oversized shared memory, B-side collectors, and early A-release mechanisms. FlashAttention-4 pairs a new algorithm with a kernel co-design tuned for NVIDIA Blackwell GPUs, removing bottlenecks and keeping tensor cores busy. It is 2.7x faster than Triton and 1.3x faster than cuDNN 9.13, enabling more intelligent capabilities at a lower cost per token for long context workloads. Together Megakernel is a hand-optimized implementation that runs an entire model in a single kernel, targeting the HBM bandwidth ceiling, achieving significant performance improvements and better unit economics. together.compile automates kernel optimization, generating an optimized kernel stack at startup with a single function call, accelerating generation and improving startup times for image and video models. The Reinforcement Learning API applies the full Together stack to RL training, optimizing rollouts and accelerating weight distribution. ThunderAgent introduces a program-aware abstraction for agentic workflows, improving throughput, reducing memory imbalance, and reclaiming resources. ATLAS-2 introduces an online training flywheel for speculative decoding, continuously updating the speculator from live traffic for improved performance and adaptation. Cache-aware prefill–decode disaggregation (CPD) separates compute-heavy prefill from latency-sensitive decoding by classifying requests by cache hit rate and routing them to dedicated tiers, improving sustainable throughput for long-context inference.

3 posts · 2026

Timeline

Feature trailActive

Mamba-3 Inference-Optimized SSM

Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal. It features a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Mamba-3 SISO outperforms Mamba-2, Gated DeltaNet, and even Llama-3.2-1B (Transformer) on prefill+decode latency across all sequence lengths at the 1.5B scale. The team also open-sourced the kernels, built using a mix of Triton, TileLang, and CuTe DSL for maximum hardware performance. The architecture includes QKNorm for training stability, removes the short causal convolution by combining simple biases on B and C after BCNorm with the new discretization-based recurrence, and adopts interleaved MLP layers. The optional pre-output projection for Mamba-3 improves length generalization performance on synthetic tasks at the slight cost of in-context real-world retrieval tasks. Hybrid models combining linear layers with self-attention enable better retrieval compared to a vanilla Transformer.

1 post · 2026

Timeline

Feature trailActive

Cache-Aware Prefill-Decode Disaggregation (CPD)

Together AI has developed Cache-Aware Prefill-Decode Disaggregation (CPD), a serving architecture that separates cold and warm workloads based on cache hit rate. CPD utilizes a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache) and a dedicated pre-prefill tier for cold requests, while normal prefill nodes handle warm requests by fetching KV blocks from the cache. This architecture aims to improve sustainable throughput and reduce time-to-first-token (TTFT) for long-context inference, delivering up to 40% higher sustainable throughput under mixed, real-world traffic.

1 post · 2026

Timeline

Feature trailActive

Coding Agent Training Dataset

Together AI continues to advance agentic coding capabilities with the release of Qwen3-Coder-480B-A35B-Instruct. This model demonstrates frontier-level performance on complex, real-world software engineering workflows, including legacy system modernization, cross-system feature development, and complex debugging. Its 480B parameters and 256K native context window (extensible to 1M) allow it to hold entire codebases in memory and autonomously execute intricate tasks. Together AI's optimized infrastructure provides production-ready deployment with zero setup, instant accessibility, and significantly faster inference, enabling developers to tackle previously intractable problems.

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

Consistency Diffusion Language Models

Together AI introduces Consistency Diffusion Language Models (CDLM), a post-training recipe that accelerates Diffusion Language Model (DLM) inference by combining consistency-based multi-token finalization with block-wise KV caching. This approach significantly reduces refinement steps and enables exact KV caching, leading to up to 14.5x latency speedups on math and coding tasks while maintaining competitive accuracy. CDLM allows for block-wise causal attention, improving hardware utilization and offering a balanced operating point for small-batch settings.

1 post · 2026

Timeline

Feature trailActive

LLM Evaluation Framework

Together Evaluations provides a fast, flexible way to benchmark LLM response quality using leading open-source judge models. It supports three evaluation modes: Classify, Score, and Compare, enabling users to define custom benchmarks and compare models. The platform integrates with serverless inference APIs and offers a Python SDK for programmatic access. This release enhances the ability to make data-driven decisions on model quality, cost, and performance, supporting the evaluation of open-source, fine-tuned, and proprietary models, including those from OpenAI, Anthropic, and Google. It also supports fine-tuned models deployed via LoRA serverless inference or Dedicated Endpoints. New recipes demonstrate how to fine-tune open-source judges to outperform proprietary models and use GEPA for automated prompt optimization.

8 posts · 2025–2026

Timeline

20252026

Feature trailActive

Text-to-Speech (TTS) Model Deployment

Together AI now offers native deployment of elite proprietary Text-to-Speech (TTS) models, including Rime's Arcana v2 and Mist v2, on dedicated infrastructure. This enables high-quality, low-latency voice generation with advanced features like deterministic pronunciation control, multilingual support, and conversational realism. The integration allows for co-location with LLM and STT workloads for a unified real-time voice agent pipeline, offering a single API, authentication, and observability surface. Mist v2 provides deterministic pronunciation control, ensuring consistent rendering of defined terms across all voices and channels, crucial for high-volume production environments. Arcana v2 offers expressive, conversational voices trained on real customer service interactions. Both models are available on dedicated endpoints with a 99.9% uptime SLA and compliance certifications (SOC 2 Type II, HIPAA ready, PCI compliant).

7 posts · 2025–2026

Timeline

20252026

Feature trailActive

Python SDK

Together AI has released Python SDK v2.0, a new, type-safe, OpenAPI-driven client. This version offers improved architecture, better type safety and developer experience, future-proofing for new features, and compatibility with modern tools like `uv`. It utilizes `httpx` for better performance and includes enhanced error handling, a modern parameter system with keyword-only arguments, and new capabilities like raw response access, streaming helpers, and code interpreter session management. The SDK also introduces beta APIs for Instant Clusters and a more precise error model.

1 post · 2025

Timeline

20252026

Feature trailActive

PyTorch Reinforcement Learning Integration

Together AI is integrating PyTorch Reinforcement Learning into its AI Native Cloud platform, providing a complete solution for building, training, and deploying agentic AI systems. This complements existing services like supervised fine-tuning, sandboxed environments, evaluation tools, model optimizations, and high-performance inference, enabling end-to-end development of advanced agentic systems. The platform now supports running TorchForge RL pipelines on Instant Clusters with distributed training and sandboxed environments.

1 post · 2025

Timeline

20252026

Feature trailActive

LLM Reasoning Instruction Following Benchmark

Together AI continues to develop and promote novel evaluation frameworks for LLMs, focusing on their ability to follow instructions throughout their reasoning process. This includes the ReasonIF benchmark, which assesses multilingual reasoning formatting and length control, and aims to improve controllability transparency and safety of LRMs by identifying and addressing failures in reasoning instruction adherence.

1 post · 2025

Timeline

20252026

Feature trailCooling

Batch API for LLM Inference

Together AI's Batch Inference API offers a streamlined UI for job creation and tracking, universal access to all serverless models and private deployments, a 3000x increase in rate limits (from 10M to 30B enqueued tokens per model per user), and typically 50% lower costs compared to real-time APIs. These enhancements make it the most economical and efficient solution for high-throughput, non-real-time workloads like large-scale text analysis, embedding generation, and synthetic data generation.

1 post · 2025

Timeline

20252026

Feature trailCooling

Batch Inference API

Together AI now offers native deployment of elite proprietary Text-to-Speech (TTS) models, including MiniMax Speech 2.6 Turbo, on dedicated infrastructure. This enables high-quality, low-latency voice generation with advanced features like multilingual streaming, emotional awareness, and rapid voice cloning, integrated alongside LLM and STT workloads for a unified real-time voice agent pipeline. The Batch Inference API has been significantly improved with a streamlined UI for easier job creation and tracking. It now supports all serverless models and private deployments, offering universal model access. Rate limits have been increased by 3000x, from 10M to 30B enqueued tokens per model per user. The API also offers lower costs, typically 50% of the real-time API for most serverless models, making it an economical choice for high-throughput workloads. Ideal use cases include large-scale batch inference on models like DeepSeek-R1-0528, leveraging NVIDIA Blackwell GPUs and optimized inference engines for top speeds.

2 posts · 2025

Timeline

20252026

Feature trailCooling

AI Security and Safety Guardrails

Together AI integrates VirtueGuard, an enterprise-grade AI security and safety model, directly into its platform. This guardrail model provides comprehensive protection against harmful outputs, compliance violations, and reputational damage with an 8ms response time, high F1 score, and low false positive rate across 12 risk categories for text, images, and audio. It offers seamless integration with a single API parameter, enterprise reliability with a 99.9% uptime SLA, automatic scaling, and continuous updates from Virtue AI's research team.

1 post · 2025

Timeline

20252026

Feature trailCooling

AI Agent Evaluation Framework

Together AI is developing and promoting novel evaluation frameworks for AI agents, focusing on their ability to predict future events. This approach, termed FutureBench, draws from real-world prediction markets and emerging news to create interesting prediction tasks grounded in actual future outcomes. It addresses data contamination issues inherent in traditional benchmarks by making predictions about future events, which are inherently unverifiable at training time. The framework operates on three levels: framework comparison, tool performance, and model capabilities, allowing for systematic evaluation of agentic systems. Initial results show agentic models performing better than simple language models, with different models exhibiting distinct information gathering strategies.

1 post · 2025

Timeline

20252026