Baseten Feature Trails

Feature trailActive

Real-time Text-to-Speech with WebSockets

Baseten's platform supports real-time text-to-speech (TTS) inference using WebSockets, enabling low-latency, bidirectional communication for streaming audio applications. This builds upon existing capabilities for AI model deployment and serving, specifically enhancing interactive voice applications by leveraging persistent connections for continuous data exchange between clients and the Orpheus TTS model. The platform now also supports NVIDIA Nemotron 3 Diarization models, offering real-time speaker labeling and diarized transcription by integrating with NVIDIA's Streaming Sortformer architecture and Parakeet V2 ASR model. This enables end-to-end diarized transcription pipelines that condition the ASR model on speaker activity features within a single forward pass, improving accuracy and temporal alignment for overlapping speech.

2 posts · 2024–2026

Timeline

20242026

Feature trailActive

Agentic Code Transformation and Model Orchestration

Baseten Chains is now generally available for production compound AI systems. Chains is an SDK for deploying performant compound AI systems to production, enabling AI builders to ship ultra-low-latency compound AI with unique hardware and autoscaling for each step. This eliminates performance bottlenecks and model orchestration headaches while keeping inference cost-efficient. Enhancements since beta include output streaming, binary IO support, subclassing for Chainlet reuse, Chains Watch for live code patching, a linter, more expressive logging, and UI improvements for managing environments.

16 posts · 2024–2026

Timeline

20242026

Feature trailActive

Baseten AI Model Deployment and Serving Platform

Baseten's AI model deployment and serving platform continues to evolve, focusing on enabling smaller, open-source LLMs to outperform larger proprietary models on specialized tasks through rigorous evaluation and fine-tuning. This approach yields significant improvements in accuracy, inference cost, and latency, while also enhancing transparency and reliability. The platform supports advanced evaluation methodologies, including granular, deterministic checks and RL optimization using evaluation h. The platform now integrates MAI-Thinking-1, a commercial-grade reasoning model with clean data lineage and post-training customization options, offering a balance between open-source control and proprietary model management.

173 posts · 2022–2026

Timeline

20222026

Feature trailActive

Kimi K3 Model API Deployment

Baseten continues to optimize its AI model deployment and serving platform, focusing on specific model performance enhancements and architectural evolution. This includes developing specialized APIs for models like GLM-5.2, Kimi K2.5, Kimi K3 and GLM-5.3, prioritizing per-user throughput for real-time agentic applications. Optimizations involve tuning infrastructure for variable, bursty workloads while maintaining tight performance SLAs. The platform supports routing different traffic profiles to specialized inference clusters, and now includes Kimi K2, a 1 trillion parameter MoE model with a novel MuonClip optimizer for stable training and enhanced agentic capabilities, deployed with optimizations for tensor parallelism, KV-cache, and load balancing.

17 posts · 2025–2026

Timeline

20252026

Feature trailActive

Voice Cloning Fine-tuning

Baseten continues to optimize its AI model deployment and serving platform, focusing on specific model performance enhancements. This includes developing specialized APIs for models like GLM-5.3-Flash, DeepSeek V4 Pro 0813, and DeepSeek V4.1 Flash, such as a 'fast' API prioritizing latency for coding and agent tasks. Optimizations involve tuning parallelism, batching, and caching strategies, alongside improvements to speculative decoding and inference engines. The DeepSeek V4.1 Flash model intro. This post introduces a new capability for fine-tuning Qwen3-TTS for high-quality voice cloning, detailing three approaches: In-Context Learning (ICL), speaker-embedding-only, and fine-tuning. It provides a technical deep-dive into the fine-tuning process, including dataset preparation using ASR with character-level timestamps, model training with talker and sub-talker loss, and speaker embedding centroid calculation. The post also discusses training optimizations like learning rate schedules and improved speaker embedding derivation.

1 post · 2026

Timeline

Feature trailActive

RBAC and Enterprise Permissions

Baseten has built a robust RBAC system from the ground up, designed to scale for enterprises. This system features a unified organization with nested teams, each having isolated resources and access controls. It supports environment-level restrictions and granular API keys scoped to specific teams and models, balancing autonomy with governance.

1 post · 2026

Timeline

Feature trailActive

Baseten Delivery Network (BDN) for Model Weight Caching

Baseten's AI model deployment and serving platform now includes the Baseten Delivery Network (BDN), an opt-in weight delivery system. BDN mirrors model weights into Baseten-managed storage at deploy time, caching them across a three-tier hierarchy (node-local disk, in-cluster peer cache, mirrored origin). This significantly reduces cold start times, even during thundering-herd scale events, by minimizing reliance on upstream providers and optimizing weight transfer. BDN also reduces costs by ensuring only a single fetch per file is triggered across the cluster, preventing bandwidth saturation during burst scaling.

1 post · 2026

Timeline

Feature trailActive

AI Model Quantization and Compression

Baseten continues to optimize its AI model deployment and serving platform by exploring advanced techniques for model compression and quantization. This includes investigating novel methods like TurboQuant, which aims to significantly reduce the memory footprint of KV caches in LLMs through mathematical transformations and optimized quantization strategies. The goal is to enable more efficient deployment and serving of large models, particularly for applications requiring low latency and high throughput. This post details the application of 4-bit quantization to the FLUX.2 [dev] image generation model, achieving a 1.6x speed improvement with minimal quality degradation by optimizing matrix multiplication kernels.

2 posts · 2024–2026

Timeline

20242026

Feature trailCooling

Multi-cloud Capacity Management (MCM)

Baseten's Multi-Cloud Capacity Management (MCM) system unifies GPU deployments across 20+ clouds and regions into a single, elastic pool. It supports Baseten Cloud, Self-hosted, and Hybrid deployment modes, offering 99.99% uptime, low latency, and compliance with data residency requirements. The system manages autoscaling, failover, and latency-aware routing, providing a globally consistent orchestration layer for inference infrastructure. MCM enables dynamic workload placement based on geographic and compliance needs. Baseten Hybrid extends this by allowing sensitive workloads to run in a customer's VPC with optional spillover to Baseten Cloud for on-demand compute, enabling flexible use of existing cloud credits and commits.

6 posts · 2024–2025

Timeline

20242026

Feature trailCooling

AI Model Inference Communication Protocols

Baseten's platform supports various communication protocols for AI model inference, including HTTP for broad compatibility and reliability, WebSockets for real-time bidirectional communication in interactive applications, and gRPC for high-performance, type-safe service-to-service communication. The choice of protocol impacts application architecture, performance, and complexity. This post details a specific implementation of real-time text-to-speech using WebSockets with the XTTS V2 model, demoing the trade-offs between HTTP, WebSockets, and gRPC for AI model inference.

4 posts · 2024–2025

Timeline

20242026

Feature trailCooling

Baseten Performance Client for Embedding Throughput

Baseten has developed and open-sourced the Baseten Performance Client, a Python library built with Rust, designed to significantly improve embedding throughput for high-volume tasks. This client addresses Python's Global Interpreter Lock (GIL) limitations by offloading network-bound operations to a Rust-based Tokio runtime, enabling true parallel request execution and maximizing multi-core CPU utilization. It offers up to 12x higher throughput and lower latencies compared to standard clients, supporting both synchronous and asynchronous usage, and general batch requests for modalities beyond embeddings.

1 post · 2025

Timeline

20252026

Feature trailCooling

Model Inference Metrics Export

Baseten's platform allows users to export model inference metrics to external observability tools, integrating with platforms like Grafana, Datadrog, New Relic, and Prometheus via OpenTelemetry. This provides a unified view of AI application performance within existing observability stacks, facilitating debugging and optimization. The platform also introduces activity logging for workspace changes and request-level LLM metrics (request size, response size, time to first byte) to aid in performance analysis and debugging. The metrics interface is now customizable, allowing users to organize and prioritize displayed metrics and directly access relevant logs from metric plots.

1 post · 2024

Timeline

20242026

Feature trailCooling

Canary Deployments

Baseten introduces canary deployments, a feature that allows for gradual traffic shifting to new model deployments over a customizable time window. This enables early detection of issues and seamless rollback, improving reliability and developer experience during rollouts. The system ramps up traffic in 10 equal steps over a specified timeframe, allowing new deployments to scale appropriately and minimizing risk to end-users.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Baseten Self-hosted Inference

Baseten continues to expand its AI model deployment and serving platform by introducing Baseten Self-hosted. This offering allows enterprises to run AI model inference directly within their own cloud environments (VPCs), providing enhanced control over data locality, compliance, and hardware utilization. It leverages Baseten's expertise in AI inference, Truss for model packaging, and offers zero-downtime deployments and updates, catering to strict data residency, security, and custom hardware needs, while also enabling the optimal use of existing cloud credits and commitments.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Asynchronous Inference Queuing and Processing

Baseten introduces asynchronous inference, a system for queuing and processing inference requests based on model capacity and priority. This addresses limitations of real-time inference, such as timeouts and traffic spikes, by enabling robust handling of long-running tasks and large volumes of requests. The system provides visibility into request status, supports retries, cancellation, and delivers results via webhooks or cloud storage. It also incorporates backpressure and heartbeat mechanisms for reliability.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Control Plane and Workload Plane Separation for Model Serving

Baseten's model serving infrastructure is built on a control plane and workload plane architecture. The control plane manages model deployment, image building, and global load balancing across multiple Kubernetes clusters (workload planes). Workload planes are responsible for actual model inference on GPUs. This separation allows for multi-cloud, multi-region deployments, regional preference, data residency compliance, and self-hosted inference while centralizing management and reducing operational overhead.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2023

Truss Live Reload

Baseten's Truss model packaging format now includes a live reload feature for development deployments, significantly reducing the development loop time for ML models from minutes to seconds. This allows for near-instantaneous updates to deployed models without requiring a full rebuild and redeploy cycle, improving developer productivity and flow state.

2 posts · 2022–2023

Timeline

20222026

Feature trails