TechPrufer
Get started
Baseten
Baseten logoBaseten · Feature trail

Baseten AI Model Deployment and Serving Platform

173 posts · Aug 2022 – Sep 2026

What this trail covers

Baseten's AI model deployment and serving platform continues to evolve, focusing on enabling smaller, open-source LLMs to outperform larger proprietary models on specialized tasks through rigorous evaluation and fine-tuning. This approach yields significant improvements in accuracy, inference cost, and latency, while also enhancing transparency and reliability. The platform supports advanced evaluation methodologies, including granular, deterministic checks and RL optimization using evaluation h. The platform now integrates MAI-Thinking-1, a commercial-grade reasoning model with clean data lineage and post-training customization options, offering a balance between open-source control and proprietary model management.

All Baseten trails
  1. LangChain trains custom models for LangSmith Engine with Baseten LoopsThis post details Baseten's collaboration with LangChain, highlighting the use of Baseten Loops for training custom models for LangSmith Engine. It emphasizes how Loops provides managed infrastructure for fine-tuning (supervised, RL, long-context) and connects training directly to Baseten's inference platform. This integration allows for rapid iteration and deployment of models, enabling LangChain to specialize models for tasks like diagnosing GitHub issues and categorizing traces by failure mode. The post showcases how Baseten's infrastructure empowers teams to shape agent behavior through custom model training and production deployment.
  2. DeepSeek-V4.1-Flash: more efficient prefill for coding agentsThis post introduces DeepSeek-V4.1-Flash, a 552B-parameter multimodal MoE model with a Causal Encoder-Decoder (CED) architecture optimized for coding agents. It highlights the model's asymmetric activation parameters (8B for prefill, 16B for decode) and improved KV cache efficiency through techniques like Compressed Sparse Attention 2 and FP4 KV caching, leading to reduced memory footprint and cost. The post also details how Baseten's Inference Stack leverages NVIDIA Dynamo and KV cache-aware routing to steer requests to replicas with relevant cached context, improving inference speed and throughput for agentic workloads.
  3. Blaxel is joining Baseten to build the future of agentic infrastructureThis post announces the acquisition of Blaxel by Baseten, integrating Blaxel's agentic infrastructure primitives (fast sandboxes, persistent storage via Agent Drive, and a rebuilt networking layer) with Baseten's AI model deployment and serving platform. This integration aims to create a unified platform for serving and training models and building long-running agents, with a focus on performance, reliability, and efficiency for agent execution.
  4. How Baseten makes pyannote’s diarization models 9.6x fasterThis post details specific optimizations applied to pyannote's speaker diarization models (Community-1 and Precision-2) to improve their inference efficiency. Key contributions include: 1. Quality-aware quantization and mixed precision (FP8, BF16, FP32, FP64) to boost throughput by 3.2x for the proprietary model with minimal DER penalty. 2. An indexing strategy for clustering high-dimensional embeddings, enabling diarization of 20 hours of audio in two minutes on a single GPU, addressing the memory explosion and curse of dimensionality for long audio. 3. Scheduling and batching optimizations, CPU-GPU memory transfer reduction, and pipeline fusion for up to 9.6x speedup in long-audio latency and 3x throughput increase. The post also highlights the trade-offs between different numerical precisions for various pipeline stages and the quadratic cost growth of naive embedding pair comparison versus an indexed clustering approach.
  5. Optimizing delta weight syncs for managed rolloutsThis post details Baseten's implementation of delta weight synchronization for managed rollouts of large AI models. It describes a system that allows trainers to publish checkpoints or deltas to object storage, which are then distributed to independent rollout clusters via the Baseten Delivery Network. A key technical innovation is the custom zstd decoder that efficiently processes XOR-encoded weight differences, avoiding the materialization and scanning of the full model. This sparse decoding approach, combined with optimized GPU update strategies that group changes by destination and serialize updates for retry safety, significantly reduces the request pause time during model updates from ~12 seconds to ~6 seconds, achieving full fleet updates in under 40 seconds.
  7. Baseten leads Coval’s voice AI benchmarkThis post details how Baseten's optimized inference stack, encompassing STT and TTS latency, orchestration, networking, autoscaling, and deployment placement, leads to superior voice AI performance. Specifically, Baseten's STT deployment is shown to be approximately 5x faster than OpenAI's while achieving the lowest Word Error Rate (WER) in Coval's benchmark. The post emphasizes that production voice AI quality is defined by the full inference infrastructure, not just model choice, and highlights the importance of minimizing latency across the entire pipeline (STT, LLM, TTS, VAD, telephony, orchestration, networking, autoscaling). Baseten's ability to tune inference for production voice AI, including reliability and performance consistency through dedicated hardware and optimized deployment architecture, is presented as a key advantage.
  8. New MCP and skill for coding agents to use BasetenIntroduced a remote MCP server and skill for coding agents to interact with the Baseten platform, enabling faster and more efficient operations. This includes deploying models, checking deployment health, and debugging. Benchmarks show an average reduction of 7.5% in token usage, cost, and time, with up to 57% reduction for operation-heavy tasks. The bundle includes a backend MCP server (api.baseten.co/mcp) wrapping the Baseten REST API, a docs MCP server (docs.baseten.co/mcp) for unauthenticated search, and a basetenlabs/baseten-skills for routing agent actions. Tools are annotated as read-only or mutating for enhanced safeguards.
  9. Best open-source models for post-trainingThis post analyzes the cost drivers for post-training and serving AI models, focusing on active parameters, total parameters, and KV cache size. It categorizes and recommends open-source models (DeepSeek-V4-Flash, GLM-5.2, Kimi K2.6/K2.7 Code, Nemotron-3-Super-120B, Qwen3 family) based on their cost tiers and suitability for text, vision, and coding-agent use cases. It details how MoE architectures, KV cache compression, and asynchronous RL frameworks impact training and inference costs.
  10. The efficient frontier of LLM inferenceThis post details inference engineering techniques for LLMs, categorizing them into those that manage tradeoffs (batch sizing, parallelism strategy, quantization) and those that move the entire performance frontier (kernel optimization, speculative decoding, disaggregation). It specifically discusses how these techniques apply to models like GLM-5.3 and Kimi K3 in agentic coding scenarios with KV cache reuse, providing a framework for understanding and optimizing LLM inference efficiency.
  11. Agentic kernels in productionIntroduced an agentic kernel development framework that automates the identification, generation, and validation of optimized kernels for production AI models. This framework addresses the gap between kernel benchmarks and production deployment by considering workload-specific configurations, downstream dependencies, higher-level computation restructuring, and seamless integration into serving engines. Achieved 42.3% end-to-end latency reduction on Qwen-Image and 15.2% on FLUX.2, along with a 5.5% increase in tok/s on MiniMax M3 through optimizations like pre-packed FP8 scales, fused QKV projection and epilogue, normalization + quantization kernel fusion, bias absorption, and CFG modulation cache.
  12. GLM 5.3: Scaling with post-training, intuitively explainedThis post details the technical advancements in improving the GLM-5.2 model to GLM-5.3 through scaled post-training compute, environment design, and infrastructure engineering. It elaborates on the application of RL algorithms (SAO with compaction) and the slime framework (SGLang and Megatron) for efficient training and inference. Specific architectural optimizations discussed include Multi-head Latent Attention (MLA) for KV cache compression, DeepSeek Sparse Attention (DSA) with IndexShare for efficient attention computation at long sequence lengths, and Multi-token Prediction (MTP) for speculative decoding. The environment design focuses on realistic, agent-first task generation with robust verifier mechanisms to prevent reward hacking. The post also introduces Multi-teacher On-policy Distillation (OPD) as a new technique within the slime framework for GLM 5.3.
  13. The two AI gateway patterns in production inferenceThis post introduces and details the two dominant AI gateway patterns in production inference: access gateways (application-side, many providers) and serving gateways (customer-facing, own models). It elaborates on the serving gateway's request processing flow, covering identity establishment, serving path selection (mentioning KV cache locality as an optimization), service protection mechanisms (rate limiting, concurrency, overload protection), usage measurement, and auditable result production. It also outlines key evaluation criteria for AI gateways, focusing on customer model representation, tenant isolation, commercial API support, inference awareness, and deployment location tradeoffs.
  14. How to run any open model inside DeepSeek HarnessThis post details how to integrate the DeepSeek Harness with Baseten's inference APIs. It describes configuring Baseten as a custom model provider within the DeepSeek Harness, specifying the base URL and API protocol. It also highlights how the harness fetches and displays supported models from Baseten's `/v1/models` endpoint and demonstrates inspecting per-request latency and throughput signals like time-to-first-token and cache hit rate.
  15. How leading platforms ensure observability for LLM inferenceThis post details the observability stack for LLM inference, including key metrics (TTFT, TPOT, throughput, latency, KV cache hit rate), different log types (build, deploy, serve), and how traces help pinpoint performance bottlenecks in the request path. It also outlines how Baseten integrates with external observability tools.
  16. Inference engineering for DeepSeek V4 Pro 0813This post details the inference engineering work done to create a production-ready API for DeepSeek V4 Pro 0813. It highlights optimizations made to the inference stack (parallelism, KV cache allocation, prefill-decode worker ratio) to balance low latency and high throughput, adapting to evolved traffic patterns like longer sequence lengths and higher cache hit rates. It also covers changes to the frontend and chat template to adhere to new harness expectations and API specs, and the enablement of speculative decoding for higher TPS by leveraging the model's DSpark speculator.
  17. Introducing NVIDIA Nemotron 3.5 LightningThis post introduces NVIDIA Nemotron 3.5 Lightning, a 30B MoE model with 3B active parameters, distilled for agentic use cases. It highlights the model's ability to achieve 4x higher throughput and 30% lower time to task completion compared to other open models. The post details the fine-tuning process of Nemotron 3.5 Lightning by CodeRabbit using Baseten Training, including supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards. The fine-tuned adapter, loaded as a rank-16 LoRA, demonstrates improved accuracy, reduced cost, and fewer output tokens on Baseten Dedicated Inference.
  18. Introducing NVIDIA Nemotron 3.5 ASR StreamingThis post introduces the integration of NVIDIA Nemotron 3.5 ASR (English and multilingual variants) into the Baseten Model Library, enabling low-latency, production-ready speech recognition. It details the cache-aware FastConformer-RNNT architecture, performance benchmarks on a single H100 (up to 100 concurrent real-time streams over WebSocket, higher via gRPC), and comparative accuracy metrics (WER) on English and multilingual datasets. The post highlights the benefits of NVIDIA NIM for cache-aware inference and mentions the possibility of fine-tuning using Baseten Training.
  19. Fine-tuning Qwen3-TTS for high-quality voice cloningThis post details the process and benefits of fine-tuning the Qwen3-TTS model for high-quality voice cloning. It contrasts three approaches: In-Context Learning (ICL), speaker-embedding-only, and fine-tuning, highlighting the trade-offs in prefill overhead, speaker consistency, and time to first audio (TTFA). A detailed fine-tuning recipe is provided, covering dataset building using ASR with character-level timestamps to create utterance-level pairs, and model training with a multi-codebook approach using cross-entropy loss (talker and sub-talker components). The training process incorporates linear warmup and cosine learning-rate decay for better adaptation. The post demonstrates that fine-tuning, even on a limited dataset (1.5 hours), offers operational advantages like no reference audio requirement at inference and stable latency at scale, with improved expressiveness and prosody compared to zero-shot methods. It also touches upon future improvements using RL reward signals and automated metrics for evaluation.
  20. 22,580: GPT-2 to Kimi K3, explainedThis post details the architectural evolution from GPT-2 to Kimi K3, highlighting key innovations in LLM inference. It explains the inefficiency of decoder-only generation without caching and introduces the KV cache as a solution to avoid recomputing projections for previous tokens, noting its potential to become a memory-bandwidth bottleneck. The post then introduces linear attention as an alternative to softmax attention, which uses a feature map (ELU+1) to make Q and K vectors reassociable, allowing the growing set of K and V vectors to be folded into a fixed DxD state. This shifts the per-step cost from O(ND) to O(ND^2) when N is much larger than D, reducing memory traffic. It also discusses DeltaNet (Fast Weight Programmers) as a mechanism to address the information interference that occurs in linear attention when the state exceeds its effective capacity due to additive updates, enabling models to dynamically interact with memory contents and selectively decide which key-value associations to keep or delete.
  21. How to run Kimi K3 in any harness: routing with Baseten SwitchThis post introduces Baseten Switch, a local Mac application written in Go, that acts as a router between developer harnesses (like Claude Code and Codex) and AI models. It enables dynamic switching between native closed-source models and open-source models hosted on Baseten, facilitating A/B testing of model performance, cost, and usage directly within the developer's workflow. Key features include live model switching without session restarts, detailed metric capture (cost, tokens per second, TTFT), and configurable routing rules for orchestrator and sub-agent levels. The post also outlines recommended patterns for using open models for orchestration and sub-agents, and provides setup instructions for popular harnesses.
  22. Introducing GLM-5.2 FastIntroduces GLM-5.2 Fast, a new Model API tier for GLM-5.2 weights, optimized for per-user throughput for real-time agentic applications. This tier is designed to handle variable, bursty workloads with consistent throughput and low latency. It maintains ease-of-use with OpenAI-compatible API endpoints and a pay-per-token model. The post provides a Python code snippet demonstrating how to switch to the Fast endpoint by changing the model slug.
  23. How to choose an AI model: lessons from Notion and GammaThis post details strategies for choosing and integrating AI models, emphasizing that a single, model-agnostic harness is suboptimal. It advocates for modifying native harnesses per model to preserve their specific behaviors and context management. The post highlights the engineering investment required for model switching, enabling flexibility for reliability, cost optimization, and faster adoption of new models. It introduces A/B testing with real users as a crucial method for evaluating subjective model outputs, complementing automated evals. The core message is to select models based on specific workflow requirements and cost-effectiveness rather than leaderboard rankings. It also notes the increasing parity of open-weight models with closed ones, making them viable for diverse markets and offering greater control over pricing. Finally, it frames fine-tuning not just for margin but for market expansion, with RL being a powerful tool for task-specific optimization, contingent on explosive product-market fit.
  24. How to optimize LLM inference speed and reduce costs in productionThis post details several techniques for optimizing LLM inference speed and reducing production costs, including continuous batching, speculative decoding (draft-target, Medusa, EAGLE-3, n-gram speculation), KV cache optimizations (KV cache-aware routing, CPU offloading), quantization (mixed-precision, rotation-based, curvature-based), kernel optimizations (kernel fusion, attention kernels, asynchronous compute, PDL), intelligent request routing (geo-aware load balancing, LoRA-aware routing), and topology-aware parallelism (tensor parallelism, expert parallelism). It explains the technical mechanisms behind each technique and their impact on latency and throughput.
  25. H100 vs. H200 GPUsThis post details the technical comparison between NVIDIA H100 and H200 GPUs, focusing on their specifications relevant to AI inference. It explains how node-level VRAM (e.g., 640GB for 8x H100 SXM vs. 1128GB for 8x H200 SXM) dictates model fit, using GLM 5.2 (744B parameters) as an example requiring H200 for its weight size and KV cache headroom. The post also covers Multi-Instance GPU (MIG) for partitioning GPUs to serve smaller models cost-effectively and discusses async programming techniques (leveraging Tensor Memory Accelerators) to overlap data loading and computation, thereby improving GPU utilization and throughput. It provides guidance on choosing between H100 and H200 based on model size, traffic volume, and budget, highlighting H100 for smaller models/sporadic traffic and H200 for very large models and long-context inference.
  26. GLM 5.2 with visionThis post details the technical process of adding vision capabilities to the GLM 5.2 language model. It describes using a pre-trained vision tower from Kimi K2.6 and training only the vision projector to map its output to GLM's embedding space. The process involved two stages: Supervised Fine-Tuning (SFT) on image-question-answer pairs, observing 'grokking' in the loss curve around 900 steps, and Reinforcement Learning (RL) to restore reasoning capabilities. The authors highlight the surprising generalization of the model, identifying people not present in the training data. They also detail the training parameters (batch size 64, learning rate 5e-4) and the observation of rapid reward curve improvement during RL.
  27. Real-time video generation inference on BasetenThis post details significant performance optimizations for real-time video generation inference on the Wan 2.2 model. Key contributions include: 1) Timestep distillation reducing generation steps from 40-80 to 4, achieving ~20x speedup using Distribution Matching Distillation (DMD2) and sequence parallelism. 2) Custom kernel engineering for Self-Attention bottlenecks, resulting in a 1.58x performance improvement for Video Sparse Attention (VSA) and kernel fusion (Adaptive LayerNorm modulation, RMS Norm, Gated Residual Add) to reduce launch overhead. 3) NVFP4 quantization for linear layers, providing ~4x higher tensor core throughput compared to BF16. These optimizations collectively yield a 53.6x speedup and a 31x cost reduction. The post also outlines scalable inference infrastructure improvements, including optimized cold starts using the Baseten Delivery Network and advanced queuing, and the implementation of custom content guardrails with a trained classifier to prevent NSFW outputs.
  28. Fast, accurate retrieval with NVIDIA Nemotron 3 EmbedThis post introduces the availability of NVIDIA Nemotron 3 Embed 8B and 1B models on the Baseten platform. It details the tradeoff between retrieval quality and indexing speed, highlighting how the 8B model offers leading accuracy while the 1B model provides a balance with reduced cost and latency. The post explains the use cases for these models in agentic retrieval and code retrieval, and mentions their availability for Dedicated Inference and fine-tuning on Baseten Training. It also notes a partnership with turbopuffer to make these models available through their native embeddings.
  29. Meet Inkling: Thinking Machines Lab's new customizable modelThis post details the deployment of Thinking Machines Lab's Inkling model (975B parameters, multimodal, MoE architecture with 41B active parameters and a 1M token context window) on the Baseten Platform. It highlights how Baseten's Inference Stack, including autoscaling and the Baseten Delivery Network (BDN), enables fast cold starts (2-3x faster) for large models. It also emphasizes the role of Multi-cloud Capacity Management (MCM) in ensuring availability and low latency by pooling GPUs across 20+ clouds, allowing workloads to shift to alternative clouds if capacity is constrained. The post also mentions the integration with vLLM for optimized serving.
  30. Introducing Step 3.7 Flash: multimodal reasoning at scaleThis post details the integration of Step 3.7 Flash, a 198B parameter sparse MoE vision-language model, into the Baseten Model Library. It highlights the hardware-efficient configuration achieved through FP8 quantization, enabling deployment on 4xH100 GPUs instead of the previously recommended 8xH200s. The post explains how MoE architecture, hybrid attention, and speculative decoding contribute to fitting the model and serving production traffic within this reduced hardware footprint, leading to lower cost per token and smoother autoscaling.
  31. Building with NVIDIA Nemotron 3 Ultra and LangChain Deep Agents Code on BasetenThis post details the integration of NVIDIA Nemotron 3 Ultra with LangChain Deep Agents on Baseten. It explains how LangChain's model-specific harness profiles, when applied to Nemotron 3 Ultra, enable it to achieve top open-model agent accuracy on the LangChain Deep Agents evaluation suite at a substantially lower cost. The setup involves installing LangChain Deep Agents Code with Baseten support and configuring the Baseten API key. The post provides a concrete example of building a simple agent for reviewing workflows and flagging compliance errors, demonstrating the use of file-reading and file-writing tools within the agentic workflow. The key technical contribution is showcasing how to leverage existing Baseten model APIs and LangChain harness profiles for advanced agentic applications without requiring model fine-tuning or custom serving infrastructure.
  32. H100 vs. H200 vs. B200: which GPU should you use?This post analyzes the trade-offs between NVIDIA H100, H200, and B200 GPUs for AI inference. It details how SXM form factor and NVLink impact performance, the role of VRAM in fitting large models (e.g., GLM 5.2), and the benefits of MIG for smaller models. It also explains how FP4 precision in the Blackwell architecture (B200) increases inference throughput by reducing memory footprint and improving compute efficiency, and how asynchronous programming (Tensor Memory Accelerator, TMEM) keeps GPUs from waiting on data. The post provides guidance on choosing the right GPU based on model size, traffic volume, and budget, highlighting specific use cases for each GPU.
  33. How to run GLM-5.2 in any harnessThis post details the integration of the GLM-5.2 model into three popular developer harnesses: Claude Code, Codex, and Deep Agents CLI (LangChain). It provides concrete configuration steps, including setting API keys and base URLs for Baseten's inference endpoint (`https://inference.baseten.co` and `https://inference.baseten.co/v1`), and specifying the model identifier (`zai-org/GLM-5.2`). The post highlights the ability to generalize this setup to other harnesses by providing the API key and endpoint.
  34. AI training vs. inference: what's the difference?This post clarifies the fundamental differences between AI training and inference, detailing their distinct hardware needs, optimization techniques, costs, and performance metrics (TTFT, TPOT, throughput, latency). It outlines the model lifecycle from pre-training to serving and highlights Baseten's role in maximizing hardware efficiency through GPU batching, multi-cloud management, and applying advanced optimization techniques like custom kernels, KV cache optimization, and speculative decoding for efficient inference.
  35. Live draft model training for speculative decodingIntroduced a distributed training pipeline for live draft model training that extracts hidden states directly from live inference to train draft models on the fly. This pipeline addresses storage overhead, compute bottlenecks, and alignment drift associated with traditional draft model training. Key technical implementations include using mmap-backed arrays for buffering training data, offloading network communication and data buffering to a background process with CUDA event synchronization, and utilizing UCXX for RDMA transfers and Trio for structured concurrency and error handling in a distributed cluster.
  36. NVIDIA BioNeMo Agent Toolkit on BasetenThis post details the integration of NVIDIA's BioNeMo Agent Toolkit with the Baseten platform. It highlights how Baseten's inference infrastructure supports BioNeMo NIM microservices and skills, enabling the creation of scientific AI agents. The post describes the architecture of BioNeMo skills, which translate HTTP endpoints into agent-executable instructions, and showcases examples of scientific workflows like protein folding and drug discovery pipelines orchestrated by agents. It emphasizes Baseten's role in providing scalable, production-ready infrastructure for these complex scientific AI workloads, abstracting away the complexities of model serving and GPU management for life science researchers.
  37. How we built the world’s fastest API for GLM-5.2This post details the technical optimizations implemented to achieve industry-leading inference performance for the GLM-5.2 model on Baseten's platform. Key contributions include: implementing shared DSA weights in the custom inference engine, performing in-house NVFP4 quantization for Blackwell GPUs, leveraging NVIDIA Dynamo for KV-aware routing to improve cache hit rates and reduce prefill burden, disaggregating prefill and decode workloads for a 2x increase in TPS, and utilizing GLM-5.2's Multi-Token Prediction heads for speculation to further boost throughput.
  38. The best open-source large language models (LLMs)This post provides a technical deep dive into nine leading open-source LLMs, evaluating their architectures, performance benchmarks, and suitability for production deployment on Baseten. It details specific technical innovations within each model, such as DeepSeek's CSA/HCA for KV cache reduction and mHC for training stability, Gemma 4's interleaved attention, GLM-5.3's SAO for asynchronous optimization, GPT OSS 120B's integration with TensorRT-LLM and EAGLE-3 speculative decoding, Kimi K3's AttnRes and KDA/MLA for long context, and MiniMax M3's Sparse Attention. The analysis highlights tradeoffs in context window size, inference speed, memory usage, and cost-effectiveness for different use cases like agentic coding and long-context reasoning.
  39. Rolling deployments for zero-downtime model updatesIntroduced rolling deployments for zero-downtime model updates. Implemented two provisioning modes: `max_surge` (temporarily exceeds steady-state capacity) and `max_unavailable` (stays within original resource footprint). Developed a durable workflow orchestration for stateful deployments, including automatic retries and pause/resume semantics. Integrated autoscaling coordination to manage load spikes during rollouts. Added mid-deploy controls (pause, resume, cancel, force cancel, force roll forward) and a configurable stabilization period.
  40. Mercury 2, the first reasoning diffusion LLM, is now on BasetenThis post introduces the integration of Inception's Mercury 2, a diffusion LLM (dLLM), onto the Baseten platform. It highlights the dLLM architecture's inherent speed advantage over autoregressive models by drafting and refining outputs in parallel passes. The post details how Baseten's infrastructure, including NVIDIA GPU support (Hopper H100, Blackwell), Baseten Frontier Gateway for rate limiting and routing, and autoscaling, enables production-grade deployment of these dLLMs. It showcases real-world results from Augment Code, demonstrating an 82% reduction in compaction latency and a 90% cost reduction by using Mercury 2 for specific subagent tasks within a multi-model system.
  41. Introducing NVIDIA Nemotron 3 Ultra: The Nemotron 3.x family is here!This post introduces NVIDIA Nemotron 3 Ultra, a new model architecture that addresses the performance degradation of transformer models in long-running agentic workflows. Nemotron 3 Ultra employs a hybrid architecture, predominantly using Mamba layers instead of pure attention mechanisms. This design choice linearly scales inference cost with context length, contrasting with the quadratic scaling of transformers, thus enabling agents to maintain speed over hundreds of reasoning steps. The model is open-source with open weights, data, and recipes. The post highlights the trade-off: Mamba's state-space approach offers efficient summarization, while a smaller number of attention layers provide precision when needed. This architectural innovation is directly integrated into Baseten's platform, making Nemotron 3 Ultra available for use in agentic applications. Additionally, Nemotron 3.5 ASR and Nemotron 3.5 Content Safety models are now available on Baseten.
  42. MAI-Thinking-1 is coming to BasetenThis post announces the integration of Microsoft AI's (MAI) MAI-Thinking-1 model into the Baseten platform. It highlights MAI-Thinking-1's characteristics: clean data lineage, commercial-grade design with data privacy, and efficiency (35B active parameters). The post emphasizes that Baseten will host and operate the model, allowing customers to fine-tune it using their own data, with the fine-tuned checkpoints being customer-controlled. It also mentions the synergy with Baseten's Loops SDK for training and iterative workflows.
  43. NVIDIA Cosmos 3: Robots finally take overThis post introduces NVIDIA Cosmos 3, a world foundation model for physical AI, and details its deployment on Baseten. It explains two primary usage patterns for robotics: direct control via the `policy` mode and data generation for training smaller onboard models using `inverse_dynamics`, `forward_dynamics`, `text2video`, and `image2video` modes. The post highlights Cosmos 3's ability to generate physically plausible data, contrasting it with creative video models, and emphasizes its role in overcoming the data collection bottleneck in robotics by transforming real-world or synthetic video into labeled training data. It also provides example API requests for `policy` and `inverse_dynamics` modes, showcasing parameters like `model_mode`, `vision_path`, `action_chunk_size`, and `raw_action_dim`. The post notes that Cosmos 3 Nano can be deployed on Baseten with specific cold-start and warm inference times.
  44. Timestep distillation: 2.5x faster FLUX.2 image generationThis post details the application of timestep distillation to FLUX.2 for faster image generation. It explains the core concepts of Distribution Matching Distillation (DMD) and its evolution to DMD2, including distribution-level matching, the two-timescale update rule (TTUR), and the use of a GAN discriminator. Engineering challenges and solutions for scaling DMD2 to FLUX.2 are discussed, focusing on memory optimization through mixed-precision training (BF16), FSDP, and activation checkpointing, and the use of NVIDIA's FastGen framework.
  45. Powering Inference for the Continual Learning EraThis post details the infrastructure built with Trajectory to support continual learning. Key contributions include a pipeline that merges LoRA adapters into base models, performing architecture-aware correctness checks (handling fused projections, MXFP4 quantization, and MoE expert broadcasting). The merged model is packaged with Truss. The serving runtime applies optimizations like quantization and RoPE scaling, with automatic validation of checkpoints against these constraints. A routing layer treats new checkpoints as experimental variants, splitting traffic and partitioning telemetry for comparison. Every request is logged with provenance, closing the loop back to training.
  46. Sub-second image generation with Flux.2 and Qwen-ImageThis post details optimizations for image generation serving, specifically for Flux.2 [dev] and Qwen-Image models on NVIDIA Blackwell and Hopper GPUs. Key technical contributions include: hardware-aware quantization (FP4 on B200, FP8 on H100), memory optimizations, optimized attention kernels, specialized element-wise kernels, and runtime-level serving improvements. These optimizations resulted in significant latency reductions, achieving sub-second inference for Flux.2 [dev] on B200 (0.98s) and Qwen-Image on B200 (0.87s), outperforming baseline configurations by up to 5.4x and 4x respectively. The post also outlines the benchmarking methodology and common image generation inference parameters supported by the optimized stack.
  47. Cost-efficient, high-performance TTS with Qwen3-TTSThis post details optimizations for cost-efficient, high-performance Text-to-Speech (TTS) using the Qwen3-TTS model with vLLM-Omni. Key contributions include achieving $3-$4 per million characters through techniques like disaggregated acoustic token generation and decoding for pipelining requests, AR-stage CUDA graphs for faster transformer inference, reduced GPU to CPU syncs by switching to vLLM's `uniproc` executor, speaker embedding caching for zero-shot voice cloning, and dynamic frame accumulation to balance latency and throughput. The post also highlights contributions back to the vLLM OSS community.
  48. Baseten Loops SDKIntroduces Baseten Loops, a new SDK specifically designed for scaling Reinforcement Learning (RL) post-training. It addresses challenges in current RL infra by abstracting hardware, simplifying the train-to-deploy loop (including LoRA merging and quantization), enabling asynchronous and bounded off-policy RL for seamless training/sampling overlap, supporting long-context RL (131K+ sequence lengths), and providing predictable performance through dedicated infrastructure. It aims to make frontier-scale RL accessible to ML teams.
  49. DFlash: 3x faster LLM inferenceThis post details Baseten's implementation of DFlash, a novel speculative decoding technique for LLM inference. It explains how DFlash achieves up to 3x speedups by predicting multiple tokens in parallel using block diffusion and bidirectional attention, surpassing autoregressive methods like EAGLE and improving upon vLLM's DFlash implementation. The post covers Baseten's custom forward pass mechanism for preparing the KV cache and producing output tokens, guided decoding support, and presents benchmark results across GSM8k, MATH-500, and NVIDIA's Nemotron datasets. It also outlines the process for training DFlash draft models, including input requirements, loss weighting, and attention mask structure.
  50. Inference Infrastructure for AI Labs | BasetenIntroduced the Baseten Frontier Gateway, a managed API gateway built on top of Baseten Dedicated Inference. This gateway provides a production-grade, multi-tenant inference API for model labs, handling authentication, authorization, federated API key management, per-user rate and usage limits, and billing/metering. It is co-located with inference infrastructure to minimize latency and integration overhead, allowing labs to serve models under their own domain.
  51. NVIDIA Nemotron 3 Nano Omni: Build multimodal agents on BasetenThis post announces day-zero support for NVIDIA Nemotron 3 Nano Omni on the Baseten platform. It highlights the model's capabilities for multimodal reasoning across audio, video, images, and text within a unified loop, emphasizing its efficiency through latent MoE design, 3D convolutional layers, and efficient video sampling. The post details how Baseten's platform accelerates enterprise AI initiatives with high-performance inference, Multi-cloud Capacity Management (MCM), expert engineering support from Forward Deployed Engineers (FDE), and robust enterprise security features. It also mentions the underlying Baseten Inference stack utilizing NVFP4, TensorRT-LLM components, Dynamo, and the Baseten Speculation Engine on NVIDIA Blackwell GPUs.
  52. How to train custom EAGLE-3 heads for speculative decodingThis post details the integration and training process for EAGLE-3 speculative decoding heads within Baseten's AI model serving platform. It provides a technical deep dive into the EAGLE-3 method, including concepts like TTT-length and number of draft tokens, and offers specific guidance on dataset preparation (regenerating outputs with the target model), hyperparameter tuning (learning rate scaling with model size, greedy decoding preference), training workflow steps, monitoring metrics (loss, accuracy), evaluation benchmarks (1.5-2.5x latency improvement), deployment considerations, and debugging common training and inference issues. The contribution is the practical application and explanation of how to train and leverage custom EAGLE-3 heads for significant LLM inference latency reduction.
  53. How the Baseten Delivery Network (BDN) makes cold starts fastIntroduced the Baseten Delivery Network (BDN), an opt-in weight delivery system for model deployments. BDN reduces runtime dependency on upstream providers by mirroring model weights into securely managed storage at deploy time. It caches weights across a three-tier hierarchy: node-local disk, an in-cluster peer cache backed by a consistent hash ring, and mirrored origin. BDN coordinates downloads to ensure fast cold starts during scale events and reduces costs by offloading weight transfer from billable GPU time and minimizing egress costs through deduplication of unchanged weight data. The system addresses brittleness in weight transfer at scale by mitigating slow pulls, upstream fragility, and thundering herd issues through its layered caching and push-time mirroring approach.
  54. Sub-3 millisecond named entity recognition (NER) inferenceThis post details the technical optimizations made to Baseten Embeddings Inference (BEI) to achieve sub-3 millisecond inference for Named Entity Recognition (NER) models. Key contributions include: a Rust-based web server and tokenizer to minimize per-request overhead, Nginx configuration for a memory-backed and lightweight request path, HTTP/2 connection pooling in the Baseten Performance Client to reuse connections, and the introduction of a `/predict_tokens` default route for NER models. It also provides an example YAML configuration for deploying a BERT-family NER model and a Python script for retrieving token-level predictions.
  55. Open-source LLM training is a mess. Here is how it all works.This post provides a detailed technical overview of the open-source LLM training ecosystem, mapping its four key layers: Systems (CUDA, cuBLAS, NCCL, NVLink, InfiniBand, Triton), Core Runtime (PyTorch, Safetensors), Training Stack (Transformers, Performance Primitives like FlashAttention, Transformer Engine), and Scaling Frameworks (Megatron, DeepSpeed, FSDP2). It explains the role and dependencies of each component, offering insights into how they fit together for reliable model training and deployment. The post highlights specific libraries and their contributions, such as FlashAttention's memory efficiency and Transformer Engine's FP8 mixed-precision training.
  56. Baseten Training: an autoresearch substrateThis post introduces the concept of an 'autoresearch substrate' built on Baseten Training, formalizing the loop of human-defined research programs executed by AI agents. It details how Baseten Training's features (CLI-first, containerization, on-demand compute, observable results, single entrypoint submission) naturally support this agentic iteration, avoiding common pitfalls of unreproducible experiments. The post provides concrete examples of prompts and experimental outcomes, highlighting how this setup accelerates methodical, bounded optimization within a defined search space, shifting the researcher's role to curating the search space rather than running experiments manually. It also discusses the limitations of pure agentic iteration and the need for human intervention to drive novelty.
  57. Introducing the Baseten Delivery Network: Fast cold starts for big modelsIntroduced the Baseten Delivery Network (BDN) to significantly reduce cold start times for large models. BDN employs a three-tier caching strategy (local node, peer, and mirrored origin) and single-flight downloads to mitigate the 'thundering herd' problem during rapid scaling. This architecture ensures faster and more consistent model weight downloads, improving the overall inference loop by accelerating the distribution of all deployment artifacts.
  58. NVIDIA Nemotron 3 Super for agentic AI in financial servicesThis post introduces day-0 support for NVIDIA Nemotron 3 Super, a 120B hybrid mixture-of-experts model, on the Baseten Inference Stack. It details the model's architecture, including latent MoE and multi-token prediction (MTP), and its performance benefits (over 50% faster token generation) for agentic workloads. The post highlights the model's suitability for financial services due to its open nature and strong benchmark performance across reasoning and coding tasks. It also contrasts Nemotron 3 Super with Nano and Ultra variants, outlining use cases in financial services like loan processing, fraud detection, and cybersecurity. The post emphasizes Baseten's role in providing flexible deployment options (Baseten Cloud, single-tenant, self-hosted) for sensitive financial applications.
  59. How we built the fastest GLM 5 APIIntroduced custom kernels for DeepSeek Sparse Attention (DSA) to optimize KV cache cost by using a lighting indexer to identify top-K tokens and perform sparse attention. Optimized the indexer code path by skipping it for sequences shorter than K and fusing operations for high-precision projections. Implemented a low-overhead MTP speculative decoding engine by running the draft and target models as one, minimizing host overhead. Optimized MoE dispatch kernels and quantized model weights to NVFP4 for Blackwell compatibility.
  60. Outperforming frontier models on emergency medicine documentationThis post details the development of a specialized AI model for emergency medicine documentation using iterative Supervised Fine-Tuning (SFT) and prompt mutation. It showcases how to build comprehensive evaluation frameworks with LLM-as-judge (Lumina) to address specific failure modes in complex medical text generation. The post highlights pipeline simplification by consolidating multiple LLM calls into fewer stages and replacing pattern-matching with deterministic code. It also demonstrates precise template modification techniques and the handling of high-risk diagnosis detection through rule-out criteria. The resulting model significantly outperforms general-purpose frontier models like gemini-2.5-pro in accuracy and latency for this domain.
  61. 4-Bit Quantization for Inference OptimizationThis post details the optimization of the FLUX.2 [dev] image generation model using 4-bit quantization for inference. It explains the model's architecture, identifies matrix multiplication (GEMM) as the primary bottleneck (67% of inference time), and describes the process of quantizing BFloat16 weights to FP4. The explanation includes the mathematical basis for linear mapping between unquantized and quantized values, highlighting how reduced precision increases effective bandwidth and FLOPS, leading to a 1.6x speed improvement with negligible quality loss.
  62. MiniMax M2.5: Intelligence too cheap to meter, RL process rewards, real-world productivityThis post details the integration and performance of MiniMax M2.5, a frontier MoE model. Key advancements include an efficient reinforcement learning setup with per-step process rewards to optimize task decomposition and reasoning, leading to significant improvements in token efficiency and cost-effectiveness compared to closed-source models. The model demonstrates SOTA benchmarks in coding, browser search, and tool use, and is capable of completing economically valuable real-world tasks. The platform now supports this model, making advanced AI capabilities more accessible and affordable for agentic applications and complex workflows.
  63. AI model performance metrics explainedThis post introduces and explains the three core AI model performance metrics: Time to First Token (TTFT), Tokens Per Second (TPS), and End-to-End Latency. It details how each metric impacts user experience, discusses the influence of model size and provider optimizations, and provides guidance on evaluating API providers based on performance stability and consistency. The post also highlights the inherent tradeoffs between performance, cost, and quality, and maps specific use cases to the most relevant performance metrics.
  64. Fine-tuning models, AI and Hollywood: A conversation with Oxen’s founder GregThis post details the evolution of Oxen AI's platform, which started as a version control system for ML datasets and has expanded to include fine-tuning infrastructure. The core technical contribution is the integration of dataset versioning with compute infrastructure for end-to-end model ownership. It highlights the challenges of managing large model weights (e.g., 600GB DeepSeek models) and the need to tie specific dataset versions to model weights. The post discusses customer use cases in LLMs (accuracy, speed, privacy) and diffusion models (VFX, advertising), emphasizing the importance of fine-tuning for quality and customization. It also touches on the technical challenges of character and scene consistency in AI-generated video and the role of AI in accelerating VFX workflows.
  65. How to run LLM performance benchmarks (and why you should)This post details the process of replicating the InferenceMAX LLM performance benchmark on Baseten using TensorRT-LLM. Key contributions include: provisioning 4xB200 GPUs, configuring `config.yaml` to match the InferenceMAX TensorRT-LLM image, adapting benchmark scripts to use the `/completions` endpoint and `ignore_EOS` flag, and trussing the deployment. It introduces server-side benchmarking against `localhost:8000/v1` to isolate inference performance from network variability, contrasting it with client-side benchmarking for user experience. The post also emphasizes the importance of developer experience constructs like caching (using `model_cache` for GPT-OSS-120B), server-side benchmarking, and scripting for iterative development, and discusses dataset selection strategies (production, public, synthetic data).
  66. OpenClaw + Kimi K2.5 on Baseten = frontier agent performance with OSSThis post details the integration of the OpenClaw agent framework with Baseten's Kimi K2.5 Model API. It provides a step-by-step guide for users to set up and run OpenClaw using Baseten, highlighting the cost savings and performance benefits of using Kimi K2.5 for agentic tasks compared to closed-source alternatives. The technical contribution lies in the practical guide and the demonstration of achieving high-performance agent capabilities with OSS models on the Baseten platform.
  67. The Baseten Inference Stack at NVIDIA Dynamo DayThis post details Baseten's integration and contribution to NVIDIA Dynamo, an inference framework for scaling generative AI workloads. It highlights how Baseten uses Dynamo's disaggregated serving (prefill/decode), KV cache-aware routing, and KV cache offloading to achieve significant performance gains (e.g., 50% reduction in time to first token, 34% reduction in time per output token, 61% higher throughput). The post also touches on Dynamo's support for multimodal model serving and KV cache offloading mechanisms like NIXL. Baseten actively contributes to Dynamo and related OSS projects like TensorRT-LLM, SGLang, and vLLM, hardening them for production and sharing improvements back to the community.
  68. Open-sourcing Baseten’s suffix automaton MTP acceleratorIntroduced a hybrid speculative decoding approach combining suffix automata (SA) with multi-token prediction (MTP) for LLM inference. This approach leverages SA for long pattern matching and MTP for general speedups, with a threshold to switch between them. The implementation is integrated into NVIDIA TensorRT-LLM, with SA states updated directly on the device without additional synchronization points. This results in up to 40% higher throughput or 40% lower latency on agentic coding workloads, with near-zero overhead and full GPU utilization.
  69. Wan 2.2 video generation in less than 60 secondsThis post details the development of a highly optimized inference runtime for the Wan 2.2 video generation model, achieving 3x faster inference on NVIDIA Blackwell GPUs and 2.5x faster on NVIDIA Hopper GPUs compared to the default runtime. Key technical contributions include: custom CUDA kernel optimizations for GEMM, RoPE attention, LayerNorm, and RMSNorm tailored to Wan 2.2's input shapes; bypassing serial all-to-all communication in Ulysses Sequence Parallelism (USP) using NCCL and Blackwell's asynchronous programming; and setting `offload_model` to false to maximize GPU throughput. The post also discusses benchmarking methodology, inference parameters, and ongoing research into lossy optimizations like attention caching and FP4 quantization.
  70. The fastest Whisper — with streaming and diarizationThis post details significant performance and cost optimizations for the Whisper transcription pipeline, achieving a 2x speed improvement (2400x RTF) and 78-98% cost reduction. It introduces real-time streaming transcription with configurable update cadences and consistent low latency. Additionally, it adds speaker diarization with a custom speaker assignment algorithm and a sliding-window approach for real-time, audio-only processing, achieving lower Diarization Error Rates (DER) and higher throughput than competitors. The pipeline is built on Baseten Chains, allowing for decoupled components and flexible GPU configurations.
  71. A Q&A From Inference To Training: The Inside Story Of Baseten’s Newest ProductThis post introduces Baseten's new training infrastructure, a significant expansion of its AI model deployment and serving platform. Key technical contributions include: a flexible training environment that accepts custom code, images, and repositories without heavy abstractions; persistent storage primitives to accelerate iteration by avoiding repeated data downloads; a pipeline for seamless transition of training checkpoints to inference for end-to-end deployment and evaluation; support for diverse model types beyond text-based LLMs, including audio models; and multi-node training capabilities for complex tasks like longer sequence lengths. The platform is designed for developers who need to debug, tweak parameters, and experiment with non-standard approaches. Migration is simplified by Baseten's unopinionated approach, allowing users to bring existing training pipelines with minimal code modification. The integration with Baseten's inference expertise is highlighted, with a focus on understanding inference requirements (time to first token, throughput) early in the training process. Customer feedback, particularly regarding private Docker image support, drove rapid feature development, demonstrating an FDE-driven development cycle.
  72. NVIDIA Nemotron 3 Nano: Build Agentic AI Applications on BasetenThis post announces day-zero support for NVIDIA Nemotron 3 Nano on the Baseten platform. It highlights the model's hybrid MoE architecture, its performance benefits (4x faster token generation, reduced compute, predictable cost), and its suitability for agentic AI systems. The post details how Baseten's Inference Stack, Multi-cloud Capacity Management (MCM), and enterprise security features (SOC 2 Type II, HIPAA compliant, self-hosting) facilitate the deployment and scaling of Nemotron 3 Nano for enterprise use cases, particularly in financial services and retail.
  73. Parsed + Baseten: Building Models That Touch GrassThis post announces the acquisition of Parsed by Baseten, detailing the strategic integration of Parsed's specialized model training systems with Baseten's inference and training stack. The core technical contribution is the establishment of a unified platform that facilitates continuous model improvement by feeding real production data back into training loops. This enables faster iteration cycles, greater engineering control, and the development of specialized, high-performance models. The post highlights the benefits for Parsed customers, including continued support, improved performance and cost-efficiency on Baseten's infrastructure, and real ownership of model weights and training data. It emphasizes the creation of tighter loops from production data to learning signals and gradient updates, allowing models to improve with usage.
  74. DeepSeek V3.2's path to GPT-5-level performance: sparse attention, RL at scale, and context reuseThis post details the technical contributions of DeepSeek-V3.2, focusing on three key areas: 1) DeepSeek Sparse Attention (DSA) which uses a learned indexer and token selector to reduce the O(N^2) attention cost to O(N*k), enabling efficient processing of long contexts (128k). 2) Scaled Reinforcement Learning (RL) with GRPO, incorporating stability fixes and a significant RL budget (>10% of pre-training compute), leading to qualitatively new behaviors and improved reasoning. 3) Revised thinking retention for tool use, where reasoning traces persist across tool calls, reducing token inefficiency and latency in complex agentic workflows. The post highlights the intelligence vs. efficiency tradeoff, positioning DeepSeek-V3.2 as a cost-effective alternative for high-quality reasoning tasks.
  75. Enterprise vision intelligence with Mistral AI and BasetenThis post details the deployment of Mistral Large 3 675B and Ministral 3 models on Baseten's enterprise inference platform. It highlights the challenges of deploying large models efficiently, particularly on NVIDIA Blackwell B200 GPUs, and how Baseten's multi-cloud capacity management and self-hosted inference solutions address these. The post also mentions the capability to fine-tune Ministral 3 models using Baseten Training (SFT and RL).
  76. Purpose-built LLMs for dental note-takingThis post details the development of purpose-built LLMs for dental note-taking, focusing on three tasks: converting ambient transcripts to structured notes, real-time note updates, and note alteration. Key technical contributions include: 1. Development of a comprehensive evaluation framework (Lumina) with six binary evaluators to capture domain-specific failure modes (clinical terminology, contradictions, completeness, source fidelity, structural compliance, tooth notation). 2. Application of iterative Supervised Fine-Tuning (iSFT) for efficient model training, leveraging LLM-as-a-judge evaluations for dense, token-level supervision, resulting in 6-7x better sample efficiency than standard SFT. 3. A novel synthetic data generation approach to embed complex domain knowledge (e.g., four tooth notation systems) into models by transforming and rephrasing source text to increase diversity, generating over 50,000 training examples across translation, exam-style, scenario-based, and summarization tasks. The resulting models match `gemini-2.5-pro`'s accuracy on tooth notation and outperform it on real-time tasks, while responding 5-6x faster.
  77. Train AI Models When You Want. Deploy on Ultra Performant Infrastructure. Baseten Training Is GA.This post announces the General Availability (GA) of Baseten Training. It details the core capabilities of the training platform, emphasizing its performant infrastructure, developer-first experience, and solutions to common customer pain points like limited compute access, opaque pricing, and underutilized capacity. New features for GA include an ML Cookbook with pre-built training recipes for various models (Gemma3, GPT OSS 20B/120B, Qwen3 30B, Orpheus, Whisper), UI improvements for job management and metrics, 'Resume from Checkpoint' functionality, 'Deploy from Checkpoint' for fine-tuned models, new Infiniband-backed clusters for faster distributed training, and improved logging and observability. The post highlights the ability to run multi-node jobs with persistent storage, cache models/datasets, track metrics granularly, and deploy checkpoints directly to inference endpoints.
  78. High-performance agents for financial services with NVIDIA Nemotron on BasetenThis post details the integration of NVIDIA Nemotron Nano 2 VL and Nemotron Parse 1.1 models into the Baseten platform, enabling high-performance, scalable inference for vision language and OCR tasks. It highlights the use of NVIDIA NIM microservices for deployment and Baseten's multi-cloud infrastructure, including autoscaling GPUs, active-active reliability, and independent workload planes, to support demanding enterprise workloads in financial services. The post also references Baseten's existing capabilities in model performance (e.g., GPT-OSS API with NVIDIA Dynamo), multi-cloud capacity management, forward-deployed engineering support, and enterprise-grade security.
  79. How we made the fastest GPT-OSS on NVIDIA GPUs 60% fasterThis post details the implementation of EAGLE-3 speculative decoding to improve GPT-OSS 120B inference speed by 60% on NVIDIA B200 GPUs, achieving over 650 TPS. It explains the mechanics of speculative decoding, the benefits of EAGLE-3's architecture (size, accuracy via hidden state utilization), and the integration challenges with TensorRT-LLM. The post also discusses the trade-offs between NVIDIA GPUs and custom hardware, emphasizing Baseten's ability to deliver performance, flexibility, and scale on generally available hardware with full context windows. It highlights independent benchmarking results from Artificial Analysis and OpenRouter, showcasing real-world performance.
  80. DeepSeek-OCR and the Unreasonable Usefulness of CompressionThis post details the deployment of the DeepSeek-OCR model on Baseten, focusing on the practical aspects of packaging and serving a highly compressed visual model. It highlights the use of Truss for model deployment, including specific `config.yaml` configurations for dependencies (vLLM, flash-attn) and resource allocation (H100_40GB). The post also touches on debugging build commands by examining deployment logs and referencing base images. The core technical contribution is demonstrating how to achieve performant inference for a model that leverages compression for efficiency, enabling applications like scalable training data generation, near-infinite context windows, and real-time computer-use agents.
  81. Building production AI for regulated industries with a leading digital insurerThis post details the engineering effort to build a production-ready AI system for a regulated insurance industry, focusing on compliance, accuracy, and speed. Key technical contributions include: a state-of-the-art knowledge retrieval system with dynamic chunking, hybrid retrieval (dense + sparse), and multi-stage reranking; a novel evaluation harness constructor that systematically discovers failure modes through spectrum sampling, meta-error analysis, hierarchical clustering, and evaluation synthesis; a meta-evaluation process to refine evaluation prompts and align with expert judgment; genetic optimization to generate gold training examples; supervised fine-tuning of Qwen3-32B on these gold examples, achieving performance competitive with frontier models; and reinforcement learning (GRPO) for tool use, specifically optimizing retrieval query formulation and handling ambiguous questions. The architectural decision to move beyond traditional vector embeddings by incorporating these advanced techniques is also highlighted.
  82. 2x faster inference with KV cache-aware routingThis post details the integration and performance benefits of NVIDIA Dynamo's KV cache-aware routing within Baseten's AI model serving platform. It explains how KV cache-aware routing works by directing incoming requests to model replicas with relevant cached context, minimizing recomputation. Benchmarks with Qwen3 Coder 480B demonstrate a 50% reduction in TTFT and a 34% reduction in TPOT, with P95/P99 latency decreasing by 48%/49% respectively. Throughput metrics show a 61% increase in RPS and a 62% increase in TPS. The post also mentions the flexibility of Dynamo to combine KV routing with other strategies and highlights ongoing work on KV cache offloading.
  83. From Sketch to 3D Model: Building a flower card generator with open source AIThis post details the process of packaging Autodesk's WaLa research model into a production API using Truss. It covers vendoring the model's source code, writing a model wrapper with `load` and `predict` methods, configuring Truss with resource requirements (H100 MIG GPU, system packages like OpenGL libraries) and secrets management, and deploying to Baseten for auto-scaling and caching. The post also outlines the architecture of a web application built around this API, utilizing Netlify for serverless functions, Neon database for storage, and Three.js for a 3D rendering experience.
  84. Baseten brings AI video to life on NebiusThis post details the technical implementation of running a custom text-to-video system on Nebius using the Baseten Inference Stack. It elaborates on optimizations at the Model Runtime layer, including modality-specific kernels, kernel fusion, attention kernels, continuous batching, request prioritization, and topology-aware parallelism. It also describes the Inference-optimized Infrastructure, focusing on intelligent request routing, geo-aware load balancing, SLA-aware autoscaling with fast cold starts, active-active reliability, and multi-cloud capacity management. The post discusses protocol selection (HTTPS, WebSockets, gRPC) for interactive workloads and outlines key performance metrics like P90/P99 latency and utilization. It details the request flow on Nebius and highlights the synergy between Nebius's capacity and Baseten's stack for predictable performance and uptime in video generation.
  85. Making Zed fast: A conversation with Richard FeldmanThis post details Baseten's partnership with Zed Industries to provide low-latency edit predictions for their code editor. The core technical challenge addressed is minimizing the network round trip time for AI model inference, especially for geographically distributed users. Baseten's infrastructure, with its high-powered GPUs, is leveraged to achieve this low latency, enabling Zed to integrate useful AI predictions without compromising the editor's overall speed. This highlights Baseten's capability in optimizing inference for latency-sensitive applications.
  86. How Baseten MCM, our cloud ecosystem partners, and NVIDIA drive fast, reliable inference at scaleThis post introduces Baseten's Multi-Cloud Capacity Management (MCM) system, a universal orchestration layer that pools GPUs across multiple cloud providers and regions to act as a single elastic, always-available resource. MCM's core capabilities include active-active reliability for 99.99% uptime, latency-aware routing, compliance-ready deployment, and no vendor lock-in. The system leverages a cloud partner ecosystem and accelerated compute hardware from partners like NVIDIA to deliver high-performance AI inference at scale. The post highlights how MCM enables customers to operate on global, reliable infrastructure without long procurement cycles or capacity limitations, and provides future-proof scaling for AI applications.
  87. How to fine-tune gpt-oss-120b with Baseten and AxolotlThis post details how to fine-tune the gpt-oss-120b model using Baseten's Training platform and the Axolotl fine-tuning runtime. It outlines the prerequisites, including data quality and evaluation, and provides specific configuration examples for both Baseten's `config.py` (defining compute, caching, and checkpointing) and Axolotl's YAML file (specifying base model, dataset, and training parameters like flash attention and gradient checkpointing). It also demonstrates launching the training job via the `truss train push` command and deploying checkpoints for production.
  88. Fine-tuning small open-source LLMs to outperform large closed-source models by 60% on specialized tasksThis post details how Baseten leverages its evaluation-first methodology and platform capabilities to fine-tune smaller open-source LLMs (e.g., Gemma 3 27B) to surpass the performance of larger proprietary models (e.g., Claude Sonnet 4) on specialized tasks like healthcare scribing. It introduces the concept of task-specific fine-tuning to exploit lower entropy output distributions, leading to 60% better accuracy and 50-80% lower inference costs. The post elaborates on Baseten's advanced evaluation methodology, which decomposes tasks into granular, objective checks to create a robust reward signal for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). It also highlights the benefits of mechanistic interpretability for output attribution and self-improvement.
  89. How we run GPT OSS 120B at 500+ tokens per second on NVIDIA GPUsThis post details the engineering effort to achieve state-of-the-art latency and throughput for GPT OSS 120B on NVIDIA GPUs. It covers the process of testing and benchmarking across inference frameworks (TensorRT-LLM, vLLM, SGLang), ensuring compatibility with Hopper and Blackwell GPU architectures, integrating with NVIDIA Dynamo, and implementing performance optimizations such as KV cache-aware routing and speculative decoding with Eagle. The post also discusses the trade-offs between Tensor Parallelism and Expert Parallelism for mixture-of-experts models, and the contribution of bug fixes back to open-source projects.
  90. Run Qwen3 Embedding on NVIDIA Blackwell GPUsThis post introduces Baseten Embeddings Inference (BEI) for NVIDIA Blackwell GPUs, showcasing benchmark results for the Qwen3 Embedding 8B model. It highlights 1.5x higher token per second throughput on B200s compared to BEI on H100, 3.3x higher than vLLM, and 3.6x higher than TEI on H100 for high query-throughput (500 tokens/request). For low query-throughput (5 tokens/request), BEI on B200s offers 8.4x higher throughput than vLLM and 1.6x higher than TEI. The post also links to a technical deep dive on BEI optimization with TensorRT-LLM.
  91. Voxtral vs. Whisper: Build a Voice-Controlled Smart Home AppThis post details the architectural and pretraining innovations in Mistral's Voxtral model, specifically its adapter layer for balancing audio and text embeddings and its cross-modal continuation pretraining pattern. It demonstrates how these design choices lead to lower word error rates and enable single-pass inference for multimodal tasks like voice command parsing and tool calling. The post showcases a practical implementation of Voxtral Mini in a voice-controlled smart home app deployed on Baseten Inference, highlighting the reduction in latency and complexity compared to using separate ASR and LLM models.
  92. Kimi K2 Explained: The 1 Trillion Parameter ModelThis post details the technical innovations behind the Kimi K2 model, including its mixture-of-experts architecture with 384 specialized experts and reduced attention heads, its post-training approach using synthetic agentic data from simulated tool interactions, and the MuonClip optimizer that stabilizes training by clipping attention logits. It also outlines the production deployment optimizations required for Kimi K2, such as tensor parallelism, KV-cache optimization, and load balancing, and highlights Baseten's role in making this model accessible.
  93. AI inference explained: The hidden process behind every predictionThis post provides a foundational explanation of AI inference, detailing its two stages (training and inference), the request lifecycle, and the inherent challenges of balancing speed, reliability, and cost. It introduces key concepts like inference frameworks (TensorRT-LLM, SGLang, vLLM) and infrastructure optimizations (intelligent routing, geo-aware load balancing, SLA-aware autoscaling). It also defines critical metrics for measuring inference success: latency (time to first token, total generation time), throughput (tokens per second, requests per second), and cost, emphasizing strategies like batching.
  94. Canopy Labs selects Baseten as preferred inference provider for Orpheus TTS modelsThis post details the optimization of the Orpheus TTS model for production inference on Baseten's platform, leveraging TensorRT-LLM for performance enhancements. It introduces key performance metrics for TTS models, such as simultaneous real-time streams and time-to-first-byte (TTFB), and presents benchmark results showing significant improvements (up to 3.5x more streams, <150ms TTFB). The technical details include applying FP8 quantization, chunked context, and Llama-specific CUDA optimizations via TensorRT-LLM. It also provides sample client code demonstrating efficient real-time speech synthesis calls and highlights the benefits of session re-use. The post emphasizes the cost-efficiency of using H100 MIG GPUs for Orpheus inference and the ability to scale throughput for non-real-time tasks.
  95. Day zero benchmarks for Qwen 3 with SGLang on BasetenThis post details the day-zero deployment and benchmarking of Qwen 3 using the SGLang inference framework. It outlines the Mixture of Experts (MoE) architecture of Qwen 3 235B, explaining the sharding strategies for attention (Tensor Parallelism) and sparse MoE layers (Expert Parallelism). Performance benchmarks are provided for Qwen 3 235B with SGLang, showing median time-to-first-token, tokens per second per user, and total throughput. The post also discusses optimizations for cold start times for large model weights and offers inference tips for Qwen 3, including varying temperature, leveraging agentic capabilities, and utilizing the context window.
  96. Accelerating inference with NVIDIA B200 GPUsThis post details the introduction and technical advantages of NVIDIA B200 GPUs on the Baseten platform for AI inference. It highlights performance gains such as 5x higher throughput, over 50% lower cost per token, and up to 38% lower latency for large LLMs like DeepSeek-R1. The post discusses the B200's hardware specifications, its ability to run FP4 quantization for efficient inference, and its compatibility with fast inference frameworks like TensorRT-LLM, SGLang, and vLLM. It also outlines the four key components for a high-performance production inference service on B200s: model performance optimization, distributed GPU infrastructure, model management tooling, and AI engineering expertise.
  97. Building performant embedding workflows with Chroma and BasetenThis post details the integration of Chroma, an open-source AI application database, with Baseten's inference platform to build AI-native applications like agents, RAG pipelines, and search backends. It highlights how Baseten Embedding Inference (BEI), which offers up to 2x the throughput of previous solutions, can be used with Chroma for both initial data ingestion (speed and cost savings) and real-time query embedding (low-latency). The post provides a practical guide on deploying BEI-optimized embedding models on Baseten and using the Chroma Python SDK with the `BasetenEmbeddingFunction`.
  98. The best open-source embedding modelsThis post contributes to the Baseten AI Model Deployment and Serving Platform by providing a curated list and technical details of top open-source embedding models. It details specific models like Qwen3 8B Embedding, EmbeddingGemma, Nomic Embed Code, BAAI bge-reranker-v2-m3, and AllanAI Llama 3.1 Tulu 3 8B Reward, including their parameter sizes, base models, licenses, and recommended hardware for optimal inference (e.g., FP8 on NVIDIA B200 GPUs, H100 MIG GPUs). It also elaborates on the role of embedding models in AI applications (agents, RAG, semantic search) and discusses the importance of real-world product metrics over benchmarks. A key technical contribution is the emphasis on optimizing for latency and throughput using the Baseten Embedding Inference (BEI) engine, highlighting its performance benefits.
  99. How we built BEI: high-throughput embedding, reranker, and classifier inferenceThis post details the development of Baseten Embedding Inference (BEI), an optimized inference runtime built on TensorRT-LLM. It addresses the challenges of high-throughput corpus processing and low-latency real-time querying for modern, larger embedding models. Key technical contributions include leveraging TensorRT-LLM for performance, implementing sequence packing for efficient batching without OOM errors, and optimizing for both high throughput (up to 2.05x improvement with FP8 quantization) and low latency for individual requests. The post also discusses infrastructure considerations like queueing, backpressure, and network overhead.
  100. Introducing Baseten Embeddings Inference: The fastest embeddings solution availableIntroduced Baseten Embeddings Inference (BEI), a new inference solution specifically optimized for embedding, reranker, and classifier models. BEI achieves over 2x higher throughput and 10% lower latency compared to previous industry standards, with a focus on high concurrency and rapid single responses. It is designed for memory efficiency, allowing deployment on smaller instance types while delivering superior performance. BEI is plug-and-play and integrates with the TensorRT-LLM Engine Builder, supporting various model types and deployment options (Baseten Cloud, self-hosted, hybrid).
  101. How multi-node inference works for massive LLMs like DeepSeek-R1This post details the technical challenges and solutions for multi-node inference of massive LLMs like DeepSeek-R1. It covers infrastructure aspects such as inter-node communication using high-bandwidth interconnects like InfiniBand, and model performance optimizations including tensor parallelism and expert parallelism for Mixture of Experts (MoE) models. The post explains how to overcome VRAM limitations by distributing model weights and experts across multiple GPUs and nodes to achieve low-latency, high-throughput inference.
  102. Testing Llama 3.3 70B inference performance on NVIDIA GH200 in Lambda CloudThis post details the performance testing of Llama 3.3 70B inference on NVIDIA GH200 hardware, highlighting the architectural advantages of the GH200's high-bandwidth interconnect between CPU and GPU. The experiment demonstrates that offloading KV cache to abundant CPU memory, enabled by the 450 GB/s NVLink-C2C interconnect, allows for KV cache re-use and prefill optimizations. Benchmarking against an H100 GPU using SGLang's tool and a ShareGPT dataset showed a 32% performance gain on the GH200, attributed to the larger effective KV cache capacity. The post also discusses the trade-offs between GH200 and H100/H200 configurations for serving large models.
  103. Baseten Chains is now GA for production compound AI systemsThis post announces the General Availability (GA) of Baseten Chains, an SDK for deploying production compound AI systems. Chains allows for the creation of ultra-low-latency workflows by enabling independent hardware and autoscaling for each "Chainlet" (individual subtask or processing step). This addresses challenges in model orchestration, latency, reliability, and cost associated with compound AI. Key technical contributions include the "Chainlet" abstraction, direct inter-Chainlet communication to minimize latency, and the ability to customize hardware and scaling per component. New features since beta include output streaming for reduced initial response times, binary IO for efficient serialization of numeric data, subclassing for Chainlet reuse, and Chains Watch for live code patching and rapid testing.
  104. Private, secure DeepSeek-R1 in production in US & EU data centersThis post details the engineering challenges and solutions for deploying large open-source models like DeepSeek-R1 and DeepSeek-V3 in production. It highlights the need for multi-node and multi-cluster inference due to model size (671 billion parameters) and GPU memory constraints (e.g., H100 vs. H200 GPUs). Baseten addresses these by providing specialized hardware (H200 GPUs), supporting multi-node inference, and enabling dedicated deployments within customer VPCs for security and compliance. The post also discusses enterprise-grade performance through custom optimizations on inference frameworks like SGLang and TensorRT-LLM, and a competitive cost structure, especially for distilled models.
  105. Driving model performance optimization: 2024 highlightsThis post details Baseten's 2024 advancements in AI model performance optimization. Key contributions include adopting TensorRT-LLM as the core inference framework, integrating NVIDIA's Hopper architecture (H100, H200, GH200) with FP8 precision and MIG, and implementing speculative decoding techniques. The platform now offers featureful inference servers with guaranteed structured output, function calling, LoRA inference, and optimizations for long context. Developer tooling was enhanced with an automated Engine Builder and a custom C++ inference server. Significant performance gains were achieved across various modalities, including custom LLMs, real-time AI phone calls, and Whisper ASR. A new in-house weight distribution system was developed to reduce large model cold start times to under a minute. Future work will focus on expanding speculative decoding, optimizing embedding models, supporting new GPU architectures and quantization methods, and implementing disaggregated serving.
  106. New observability features: activity logging, LLM metrics, and metrics dashboard customizationThis post introduces three new observability features: activity logging for timestamped changes to workspace and models, request-level LLM metrics (request size, response size, time to first byte) to aid in debugging latency spikes, and customizable metrics dashboards for unified views, reordering, and direct log access from metric plots. These features aim to improve monitoring, debugging, and optimization of production AI workloads.
  107. How we built production-ready speculative decoding with TensorRT-LLMThis post details the productionization of speculative decoding with TensorRT-LLM, focusing on reducing latency for code generation tasks. Key contributions include addressing inefficient batching by implementing a custom C++-based GRPC server and fixing TensorRT-LLM's request scheduler to account for KV cache re-use and chunked prefill. It also details synchronized draft-target model execution to avoid resource contention and improve TTFT, and provides patches for TensorRT-LLM bugs related to batching and draft token decoding. The post highlights the importance of these optimizations for achieving reliable speed improvements in production LLM inference.
  108. A quick introduction to speculative decodingThis post introduces speculative decoding as a technique to reduce LLM inference latency. It explains the core concept of using a smaller 'draft' model to generate candidate tokens that a larger 'target' model validates, thereby increasing the number of tokens generated per forward pass. The post details the step-by-step process, discusses limitations such as reduced throughput with larger batch sizes and increased per-token costs, and outlines factors for a good speculative decoding setup (draft model selection, fine-tuning, resource allocation). It highlights use cases like code generation where latency is critical and batch sizes are naturally small. The post also points to Baseten's integration of speculative decoding into its TensorRT-LLM Engine builder.
  109. Introducing our Speculative Decoding Engine Builder integration for ultra-low-latency LLM inferenceIntroduces a Speculative Decoding Integration for the TensorRT-LLM Engine Builder, enabling ultra-low-latency LLM inference. This feature combines a larger target LLM with a smaller 'draft' model to predict tokens more efficiently. The integration provides pre-optimized configurations and allows users to tune parameters like the number of draft tokens and GPU memory allocation. It is particularly beneficial for low-batch, latency-sensitive applications such as code generation, where it has demonstrated halved latencies without compromising output quality. The post highlights the challenges of configuring speculative decoding from scratch and Baseten's two-tiered approach to balance ease of use with control. It also notes scenarios where speculative decoding might not be ideal, such as under very high load or with already lightweight LLMs.
  110. Generally Available: The fastest, most accurate and cost-efficient Whisper transcriptionThis post details the technical optimizations made to the Whisper model for production speech-to-text workloads, resulting in a highly performant and accurate transcription pipeline. Key contributions include: 1. Voice Activity Detection (VAD) for audio chunking to remove silences and process arbitrary lengths. 2. Utilization of Baseten Chains to create a multi-step inference pipeline with independent scaling for VAD (CPU/low-power GPU) and Whisper (powerful GPU) Chainlets. 3. Optimization of the Whisper runtime by migrating from a TensorRT-LLM Python runtime to a C++ executor for true multithreading and reduced GIL overhead. 4. Transitioning from dynamic batching to in-flight batching to immediately process requests and group them on-the-fly, significantly reducing latency. These optimizations collectively achieve over 2400x real-time factor and the lowest word error rate on benchmarks.
  111. Introducing Custom Servers: Deploy production-ready model servers from Docker imagesIntroduced Custom Servers, a new feature that allows deploying model servers directly from Docker images using a YAML configuration. This bypasses the need for Truss Server for pre-packaged images (e.g., vLLM, Triton) or proprietary Docker images. Key features include a codeless developer experience, full integration with Baseten's ML infrastructure (autoscaling, GPU availability), and support for custom readiness and liveness probes. This extends the flexibility of Baseten's model deployment capabilities.
  112. Introducing canary deployments on BasetenThis post introduces canary deployments to the Baseten platform. It details how traffic is gradually shifted to new deployments in 10 equal steps over a user-defined timeframe, allowing for monitoring and rollback. The post contrasts this with manual workarounds and highlights benefits such as decreased latency, easier development loops, and a better end-user experience. Configuration can be done via the UI or REST API.
  113. Evaluating NVIDIA H200 Tensor Core GPUs for LLM inferenceThis post provides a detailed benchmark analysis of NVIDIA H200 Tensor Core GPUs for LLM inference, comparing their performance against H100 GPUs. It highlights that H200s offer significant advantages for large models (100+ billion parameters), large batch sizes, and long input sequences, demonstrating up to 3.4x higher performance in specific long-context scenarios and 47% higher performance in high-throughput batch inference. For shorter contexts and outputs, performance gains are minimal. The analysis includes methodology using TensorRT-LLM and trtllm-bench, and discusses trade-offs between H100 and H200 for cost-efficiency in inference workloads.
  114. How to build function calling and JSON mode for open-source and fine-tuned LLMsThis post details the engineering implementation of function calling and structured output for LLMs deployed on Baseten. It explains how a modified Triton inference server ('Briton') uses logit biasing to enforce valid token generation. A state machine, generated from a regex derived from an Outlines-generated schema, is serialized to Protocol Buffers and loaded into the server. This state machine efficiently masks invalid tokens by applying pre-computed token masks based on the current state, ensuring minimal performance impact. The implementation follows OpenAI API specifications for compatibility.
  115. Introducing function calling and structured output for open-source and fine-tuned LLMsThis post introduces built-in support for function calling and structured output (JSON mode) within Baseten's TensorRT-LLM Engine Builder. This feature operates at the model server level, using logit biasing within NVIDIA's Triton inference server to enforce output schemas and tool selection during inference. The implementation guarantees 100% output structure with negligible marginal latency, addressing the challenge of integrating unstructured LLM output with structured application requirements. The functionality is available for all LLMs deployed via the Engine Builder and matches OpenAI API specifications.
  116. The best open-source image generation modelThis post introduces and reviews several leading open-source image generation models (FLUX.1, Stable Diffusion 3, SDXL Lightning, Playground 2.5), detailing their strengths, weaknesses, and licensing considerations. It highlights how these models can be deployed via Baseten's platform and discusses strategies for building complex image generation pipelines using tools like ComfyUI and ControlNet, emphasizing the control and customization offered by open-source solutions compared to proprietary alternatives.
  117. How to double tokens per second for Llama 3 with MedusaIntroduces and demonstrates the use of Medusa, a speculative sampling technique that grafts additional decoding heads onto an LLM to generate multiple draft tokens per forward pass. This technique addresses the memory-bound bottleneck in autoregressive LLM inference by leveraging idle compute resources to increase tokens per second. The post details training Medusa heads for Llama 3 8B, validating output quality with MMLU benchmarks, and deploying the optimized model with TensorRT-LLM. Benchmarks show a 94%-122% increase in tokens per second compared to a baseline deployment.
  118. Introducing automatic LLM optimization with TensorRT-LLM Engine BuilderIntroduces the TensorRT-LLM Engine Builder, which automates the creation of optimized LLM inference engines at deploy time. This addresses challenges of long setup times, hardware compatibility issues for engine building, and manual packaging/deployment. The builder integrates with Truss for model packaging and Baseten's platform for autoscaling, logging, and security. It leverages TensorRT-LLM's capabilities for model specialization, quantization, in-flight batching, and LoRA swapping to achieve low-latency, high-throughput inference.
  119. Deploying custom ComfyUI workflows as APIsIntroduced the `build_commands` feature within Truss to allow arbitrary shell commands during the Docker image build process. This enables the installation of custom nodes, model checkpoints, and other dependencies for ComfyUI workflows, caching these during the build to reduce cold start times. The post details how to modify `config.yaml` to include these commands and how to adapt ComfyUI workflow JSONs for API compatibility.
  120. How to serve 10,000 fine-tuned LLMs from a single GPUThis post details the integration of LoRA swapping with TensorRT-LLM on Baseten, enabling the serving of thousands of fine-tuned LLMs from a single GPU. It explains the mechanics of LoRA, how TensorRT-LLM compiles models, and the caching strategies for LoRAs across GPU memory, host memory, disk, and network. Key technical contributions include demonstrating that LoRA swapping is compatible with in-flight batching (each request can use a different LoRA) and has minimal impact on latency (1-2ms load time from CPU memory). It also quantifies the theoretical capacity of an H100 GPU to hold over ten thousand 16MB LoRAs in its host memory.
  121. Using asynchronous inference in productionThis post introduces asynchronous inference as a new capability on the Baseten platform. It details the workflow, contrasting it with real-time inference and highlighting its benefits for handling long-running tasks, traffic spikes, and request prioritization. Key technical aspects include request queuing, priority-based processing, webhook/cloud storage output, user-configurable retry schemas, detailed model logs, in-queue time metrics, request cancellation, backpressure mechanisms, and heartbeat for stuck requests. The implementation aims to abstract complexity for a seamless developer experience.
  122. Building multi-component AI workflows at scale with ChainsThis post introduces Baseten Chains, a new framework for building and orchestrating multi-component AI workflows. Chains are composed of modular services called Chainlets, which can be linked together for real-time inference. The framework emphasizes a delightful developer experience with local testability, type-checking, and code quality enforcement. It also provides computational expressivity, allowing for flexible computational graph structures and emergent workflow definitions. Chains extend existing Truss capabilities, offering improved code organization, local execution tooling, and the ability to deploy complex workflows with autoscaling and dedicated hardware for each component.
  123. Introducing Baseten ChainsIntroduced Chains, a new framework and SDK for building and deploying compound AI systems. Chains allow for modular workflow composition using Chainlets, enabling independent GPU and CPU resource optimization per component. The framework offers extensive Chainlet customization, integration capabilities, and performance analytics. It also provides a great developer experience with features like auto-complete, static type-checking, and a simulated local execution environment. This release aims to address customer challenges related to monolithic architectures and inefficient hardware provisioning in multi-model workflows.
  124. Comparing few-step image generation modelsThis post details the technical approaches behind few-step image generation models (LCMs, SDXL Turbo, SDXL Lightning) to achieve faster inference and higher throughput. It explains the underlying techniques such as latent consistency training, adversarial diffusion distillation, and progressive distillation, and compares their trade-offs in terms of speed, image quality, and resolution. The post also discusses how these models circumvent traditional diffusion model bottlenecks by reducing iterative denoising steps.
  125. How latent consistency models workThis post introduces Latent Consistency Models (LCMs) as a method to accelerate image generation. It explains how LCMs combine latent diffusion models with direct consistency models to operate in a lower-dimensional latent space, reducing inference steps from ~50 to 2-4. This results in significantly faster inference times (e.g., 0.5 seconds for PIXART-𝝳 vs. 3.8 seconds for SDXL on A100) for high-resolution images, making real-time generation feasible. The post contrasts LCMs with traditional diffusion models and direct consistency models, highlighting their advantages in speed and quality for image generation tasks.
  126. Comparing tokens per second across LLMsThis post details how to accurately compare tokens per second (TPS) across different LLMs by accounting for tokenizer efficiency variations. It explains that different LLMs use different tokenizers (e.g., Llama 3 with tiktoken, Llama 2 with LlamaTokenizer, Mistral models with their own tokenizers) and that these tokenizers vary in efficiency depending on the input text (e.g., code vs. prose). The post provides a method for adjusting TPS metrics to reflect real-world use by curating representative inputs, running them through the tokenizers of old and new models, calculating relative efficiency, and adjusting TPS calculations accordingly. It also mentions hardware and software optimizations like H100 MIG GPUs, TensorRT-LLM, and FP8 precision as relevant for achieving performance targets.
  127. CI/CD for AI model deploymentsThis post introduces a CI/CD framework for AI model deployments on Baseten. It details a four-step process: creating a development deployment using `truss push` (which acts as a staging environment with live reload), validating the new deployment through API calls to a dedicated development endpoint, promoting the validated deployment to production via an API call that seamlessly transitions traffic, and enabling quick rollbacks by retaining previous production deployments. The post emphasizes automating these steps into a script to save time and manual effort, highlighting the challenge of validating non-deterministic AI model outputs and suggesting separating model evaluation from deployment.
  128. Continuous vs dynamic batching for AI inferenceThis post introduces and contrasts four methods of batching inference requests: no batching, static batching, dynamic batching, and continuous batching. It details the implementation and use cases for each, emphasizing continuous batching for LLMs due to their token-by-token generation and dynamic batching for models with fixed inference times. It also mentions existing opportunities for concurrency in end-to-end systems and references specific inference servers like TGI and VLLM that support continuous batching.
  129. Using fractional H100 GPUs for efficient model servingThis post introduces the use of NVIDIA H100 GPUs with Multi-Instance GPU (MIG) technology for efficient model serving. It details how MIG allows splitting a single H100 GPU into two independent serving instances, each with approximately half the compute and memory of a full GPU. The post compares H100 MIG instances to A100 GPUs, highlighting advantages such as comparable or better performance for TensorRT-optimized models, support for FP8 precision, and increased flexibility. Benchmarks for Stable Diffusion XL and Mistral 7B demonstrate these performance benefits. It also outlines when H100 MIG is suitable (e.g., smaller models, FP8, TensorRT optimization) and when full GPUs are preferred (e.g., large models, large batch sizes, unoptimized implementations).
  130. Benchmarking fast Mistral 7B inferenceThis post details Baseten's best-in-class performance for Mistral 7B inference, achieving a 130ms TTFT, 170 TPS, and 700ms total response time for 100 tokens. It elaborates on benchmarking methodologies, including the impact of batch sizes and sequence lengths on latency and throughput, and emphasizes the advantages of dedicated model deployments. The post also discusses techniques like session reuse for reducing TTFT and the importance of accurate tokenization for TPS measurement.
  131. 33% faster LLM inference with FP8 quantizationThis post details the successful implementation and validation of FP8 quantization for LLM inference, specifically for Mistral 7B on NVIDIA H100 GPUs using TensorRT-LLM. It quantifies performance improvements (33% speed increase, 8.5% TTFT decrease, 24% cost reduction) and demonstrates rigorous quality assurance through perplexity benchmarks and anecdotal evaluations, showing FP8's viability for production workloads without significant degradation. The post also discusses previous INT8 quantization challenges and the advantages of FP8's dynamic range.
  132. High performance ML inference with NVIDIA TensorRTThis post details the integration and benefits of NVIDIA TensorRT and TensorRT-LLM for achieving high-performance ML inference. It explains the role of TensorRT in the ML inference stack, how to use it for production serving, and provides performance benchmarks for popular open-source models (Mixtral 8x7B and SDXL) optimized with TensorRT. It emphasizes the importance of GPU hardware, batch size, precision, and input/output shapes for TensorRT optimization. The post also highlights Baseten's collaboration with NVIDIA and the productization of TensorRT support within Truss, their open-source model packaging framework, offering out-of-the-box features like in-flight batching for popular models.
  133. FP8: Efficient model inference with 8-bit floating point numbersThis post introduces FP8 as an efficient data format for model inference, offering an alternative to INT8 for quantizing LLMs. FP8's higher dynamic range allows for more efficient quantization of activations and KV cache, leading to faster inference and reduced operational costs, especially for smaller models, with less degradation of output quality. FP8 is supported on NVIDIA Ada Lovelace and Hopper architectures (L4, H100 GPUs).
  134. 40% faster Stable Diffusion XL inference with NVIDIA TensorRTThis post details the process of optimizing Stable Diffusion XL (SDXL) inference by 40% using NVIDIA TensorRT. It breaks down the SDXL architecture into its core components (CLIP, UNet, Refiner, VAE) and explains how each was individually optimized by exporting to ONNX, generating optimized TensorRT engines, and integrating them back into the pipeline. Performance benchmarks on A10G, A100, and H100 GPUs are provided, along with considerations for production deployment and future optimization avenues such as optimized attention and VAE implementations.
  135. Why GPU utilization matters for model inferenceThis post explains the importance of GPU utilization for cost-effective model inference, detailing how to measure compute, memory, and memory bandwidth usage. It emphasizes that increasing batch sizes is a primary method to improve GPU utilization and throughput, while acknowledging the trade-off with latency. The post also suggests considering more powerful GPUs like H100 for highly utilized workloads and points to the Baseten workspace's metrics tab for tracking these optimizations.
  136. The best open-source large language modelThis post provides a comprehensive overview and comparison of various open-source large language models (LLMs), including Llama 3.3 70B Instruct, DeepSeek-V3, DeepSeek-R1 Qwen 32B, Phi 3 Mini, Mistral and Mixtral, Qwen 2.5 Coder, and Llama 3.1. It details their technical specifications, benchmark performance, licensing, and ideal use cases for tasks such as chat, code completion, and retrieval-augmented generation (RAG). The post also discusses the role of model evaluation benchmarks and the potential for open-source models to rival closed-source alternatives.
  137. Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRTThis post details Baseten's benchmarking of NVIDIA H100 GPUs for ML inference using TensorRT and TensorRT-LLM. It quantifies performance improvements over A100 GPUs for LLMs (Mistral 7B) and image generation models (Stable Diffusion XL), highlighting gains in throughput and reductions in latency. The post explains how H100 architecture features and TensorRT optimizations contribute to these gains, and discusses cost savings achievable by migrating A100 workloads to H100s. Baseten now offers optimized model inference on H100 GPUs.
  138. How to benchmark image generation models like Stable Diffusion XLThis post details how to benchmark image generation models like Stable Diffusion XL by standardizing configurations (hardware, model version, pipeline) and defining key performance metrics: latency (total generation time), throughput (images per minute), and cost (cost per image). It highlights the importance of considering inference steps, image resolution, concurrency, and batch size to optimize for real-world use cases and make informed tradeoffs.
  139. Understanding performance benchmarks for LLM inferenceThis post provides a detailed guide on understanding and creating nuanced performance benchmarks for LLM inference. It breaks down key factors to consider, including hardware, streaming, quantizing, input/output size, batch size, percentile reporting, and network speed. It elaborates on measuring latency (time to first token, total generation time, cold starts) and throughput (requests per second, tokens per second), emphasizing the importance of output tokens per second for real-time applications. The post also discusses cost measurement (cost per minute, cost per thousand tokens) and provides an example of a high-quality benchmark definition.
  140. Faster Mixtral inference with TensorRT-LLM and quantizationThis post details the optimization of Mixtral 8x7B inference by leveraging TensorRT-LLM and int8 quantization. It demonstrates achieving a time-to-first-token below 200ms and over 50 perceived tokens per second on a single A100 GPU. The comparison highlights a 2-3x increase in tokens per second and a 30% decrease in time-to-first-token when using TensorRT-LLM over vLLM for single requests. Quantization to int8 is shown to halve inference costs with a negligible impact on model quality (0.08% perplexity increase).
  141. Playground v2 vs Stable Diffusion XL 1.0 for text-to-image generationThis post compares Playground v2 and Stable Diffusion XL 1.0 for text-to-image generation, evaluating quality, speed, and GPU requirements. It details head-to-head image quality comparisons across various prompts, noting Playground v2's consistent aesthetic and SDXL's variety. Inference speed is found to be comparable for standard 50-step runs on an A10G GPU, with both models fitting on 16GB VRAM GPUs. The post also discusses potential optimizations for Playground v2 based on SDXL's architecture and recommends specific GPUs (T4, A10, A100) based on cost and performance tradeoffs.
  142. How to serve your ComfyUI model behind an API endpointThis post details the process of deploying ComfyUI image generation pipelines as API endpoints using Baseten and Truss. It explains how to export ComfyUI workflows in an API-compatible JSON format, package them with Truss by adding the workflow JSON and linking model weights via build commands, and deploy them to Baseten. It also provides a Python script for calling the deployed API and generating images, demonstrating the use of positive_prompt, negative_prompt, controlnet_image, and seed parameters.
  143. NVIDIA A10 vs A10G for ML model inferenceThis post analyzes the performance characteristics of NVIDIA A10 and A10G GPUs for ML model inference. It details their specifications, highlighting similarities in VRAM and bandwidth, and differences in tensor core compute. The analysis proves their interchangeability for most memory-bound inference tasks by comparing ops:byte ratios to the arithmetic intensity of Llama 2 7B. It also discusses potential advantages of the A10 in batched inference and surveys other 24GB GPUs.
  144. Stable Video Diffusion now availableThis post announces the availability of Stable Video Diffusion and highlights its capabilities for generating coherent, high-quality video from a single image. It positions Baseten as a platform for deploying such advanced AI models, offering a performant, custom instance for Stable Video Diffusion deployment.
  145. GPT vs Llama: Migrate to open source LLMs seamlesslyThis post introduces a new capability for Baseten: an OpenAI-compatible ChatCompletions API endpoint that acts as a bridge to deploy and serve open-source LLMs. It details the minimal code modifications (three lines) required to switch from using OpenAI's GPT models to open-source alternatives like Llama 3.2 11B. The post explains how to deploy these models on Baseten, obtain model IDs, and configure the OpenAI Python client with a custom `base_url` pointing to Baseten's bridge endpoint. It also covers how to enable streaming inference with these open-source models. The technical delta includes the implementation of the `bridge.baseten.co` endpoint and the demonstration of its compatibility with the OpenAI client library for both standard and streaming API calls.
  146. Open source alternatives for machine learning modelsThis post details the benefits and practical considerations of using open-source ML models as alternatives to proprietary APIs. It elaborates on the advantages of independence from vendor model shifts, custom optimizations for latency and throughput (mentioning vLLM, KV caching, speculative decoding, quantization), dedicated model instances to avoid noisy neighbors and improve security/compliance, and predictable spend through direct hardware payment and autoscaling. It also provides specific recommendations for open-source LLMs (DeepSeek, Llama, Qwen), text embeddings (BAAI bge-en-icl), audio transcription (Whisper), and audio generation (Orpheus TTS, Sesame), and links to deployment resources.
  147. A checklist for switching to open source ML modelsThis post provides a checklist for migrating from closed-source to open-source ML models. It outlines steps for selecting open-source models (LLMs, text embedding, speech-to-text, text-to-speech), choosing appropriate GPUs for inference, finding relevant optimizations (TensorRT-LLM, BEI, Whisper optimization), deploying models using the Baseten model library or Truss, and integrating the deployed model endpoints. It references existing Baseten blog posts and documentation for deeper dives into specific areas like GPU guides, optimization techniques, and model endpoint integration.
  148. Pinning ML model revisions for compatibility and securityThis post introduces the concept of pinning ML model revisions to ensure compatibility and security, drawing parallels to package version pinning. It provides code examples for using the `revision` parameter in Hugging Face's `transformers` library and discusses strategies for maintaining private copies of models. The post outlines specific scenarios where pinning model revisions is recommended, such as when using `trust_remote_code` or when strict security requirements are in place.
  149. Deployment and inference for open source text embedding modelsThis post introduces BEI (Baseten Embeddings Inference), a specialized runtime for embedding models that offers significantly lower latency and higher throughput compared to previous solutions. It details the process of creating a Truss for an embedding model, configuring BEI in `config.yaml` with specific TensorRT-LLM settings, and deploying it to Baseten. The post also demonstrates running inference using an OpenAI-compatible SDK, highlighting the performance gains achieved with BEI.
  150. NVIDIA A10 vs A100 GPUs for LLM and Stable Diffusion inferenceThis post provides a technical comparison of NVIDIA A10 and A100 GPUs for LLM and Stable Diffusion inference. It details their architectural differences (Tensor Cores, VRAM, memory type), benchmarks their performance on specific models (Llama 2, Stable Diffusion), and analyzes their cost-effectiveness for both latency-critical and throughput-oriented workloads. It also discusses the use of multi-GPU instances (multiple A10s vs. a single A100) as a strategy for balancing cost and performance for larger models or higher throughput requirements.
  151. Build your own open-source ChatGPT with Llama 2 and ChainlitThis post details the integration of Llama 2 with Chainlit on the Baseten platform to create an open-source ChatGPT-like interface. It covers obtaining Llama 2 access, deploying the model on Baseten, setting up Chainlit with Baseten API keys and model version IDs, and implementing key chatbot features: using conversation history, streaming model output by setting `stream=True` and `max_new_tokens=4096` in the Baseten API request, and stripping unwanted characters like '[/INST]' from the streamed output by buffering tokens.
  152. AI Infrastructure: Build vs. Buy | BasetenThis post provides a comparative analysis of building vs. buying AI infrastructure, focusing on performance, scalability, and cost-efficiency. It details Baseten's platform benefits, including ease of getting started with zero infrastructure experience, fast inference times, autoscaling, scale-to-zero features, and cost optimization through pay-per-compute models. It also highlights Baseten's self-hosted deployment option, dedicated forward-deployed engineer support, and security certifications (SOC II Type 2, HIPAA). The post includes insights from Chris Albon of the Wikimedia Foundation on strategic infrastructure investment.
  153. Build a chatbot with Llama 2 and LangChainThis post details how to build a chatbot using Llama 2 and LangChain, demonstrating the deployment of Llama 2 on the Baseten platform. It covers setting up the environment, obtaining Llama 2 access, deploying the model on Baseten, and integrating it with LangChain's LLMChain for conversational AI. The post highlights the use of Baseten's infrastructure for performant, scalable, and cost-efficient ML model serving, specifically for LLMs, and shows how to leverage Llama 2's context window within a LangChain application.
  154. Deploying and using Stable Diffusion XL 1.0This post details the deployment and usage of Stable Diffusion XL 1.0 on the Baseten platform. It outlines the hardware requirements (A100 GPU) and expected inference times (8-12 seconds per image). It provides instructions for manual deployment using Truss, including cloning the repository, pushing the model, and invoking predictions with example prompts. It also introduces the Stable Diffusion Refiner model and its `use_refiner` parameter for improved detail generation.
  155. Model autoscaling features on BasetenThis post introduces and details Baseten's model autoscaling features. It explains how users can configure minimum and maximum replicas to control costs and handle traffic bursts, with excess traffic being queued. The concept of 'scale to zero' is explained, where models can be put to sleep after inactivity. Specific autoscaling settings are detailed: autoscaling window (timeframe for analyzing traffic), scale-down delay (additional time before scaling down), and concurrency target (requests per replica). The post also highlights Baseten's fast cold start times, citing 9 seconds for Stable Diffusion XL on an A100, and provides load test graphs demonstrating the autoscaling features in action.
  156. Three techniques to adapt LLMs for any use caseThis post introduces three techniques for adapting LLMs: prompt engineering (zero-shot, few-shot, and explicit instructions), retrieval-augmented generation (RAG) using vector databases for context, and fine-tuning to modify model behavior. It details the use cases, costs, and trade-offs of each method, guiding users to start with prompt engineering and progress to RAG or fine-tuning as needed. The post emphasizes that prompt engineering is low-cost and effective for many tasks, RAG offers moderate cost and unlocks data-aware use cases, and fine-tuning has a high upfront cost but low ongoing cost for customized behavior.
  157. Understanding NVIDIA’s Datacenter GPU lineThis post provides a technical guide to understanding NVIDIA's datacenter GPU lineup, focusing on deciphering GPU names (architecture and tier) and comparing different cards for ML model training, fine-tuning, and serving. It details how to infer a card's capabilities from its alphanumeric identifier and offers examples of comparisons between various GPU models (T4 vs L4, A10 vs A100, K80 vs T4) to aid in hardware selection for AI workloads.
  158. Comparing GPUs across architectures and tiersThis post provides a technical guide for comparing datacenter GPUs across architectures and tiers. It details key metrics for evaluating GPU performance and cost-effectiveness for ML workloads, including core count (CUDA, Tensor, RT cores), FLOPS (at various precisions), VRAM (capacity, bus size, clock speed, GDDR vs. HBM), and TDP. It emphasizes the importance of comparing FLOPS at the same precision and core type for accurate apples-to-apples comparisons and highlights VRAM capacity as a critical factor for model size limitations during inference. The post also references existing resources for deeper dives into specific GPU comparisons.
  159. Comparing NVIDIA GPUs for AI: T4 vs A10This post provides a technical comparison of NVIDIA T4 and A10 GPUs for AI model deployment and serving. It details key specifications like CUDA cores, Tensor cores, and VRAM, and discusses factors such as price-to-performance, vertical vs. horizontal scaling, and availability. Specific use cases like Whisper for T4 and Stable Diffusion XL for A10 are presented, along with pricing information on Baseten. The post aims to help engineers choose the most cost-effective and performant GPU for their generative AI workloads.
  160. Serving four million Riffusion requests in two daysThis post details how Baseten handled a surge of 4 million Riffusion requests in two days by scaling beyond default autoscaling limits. Key technical contributions include implementing a caching layer using a Postgres table to store and retrieve common model responses, thereby reducing unnecessary model invocations. It also highlights the strategic use of a mix of spot and on-demand instances for cost-effective scaling and discusses trade-offs accepted for a hobby project, such as slightly higher latency and error rates, to manage costs.
  161. Build with OpenAI’s Whisper model in five minutesThis post details the process of packaging and deploying OpenAI's Whisper ASR model using Baseten's Truss library. It covers installing Truss, initializing a new Truss project, configuring dependencies (including a Git+HTTPS requirement for the Whisper repository and system packages like ffmpeg), and specifying GPU resource requirements. The core `predict` function is shown, along with `preprocess` and `postprocess` functions for handling audio input from a URL. The deployment process using `truss push` and prediction via `truss predict` is also demonstrated.
  162. Why we built and open-sourced a model serving solutionIntroduced Truss, a standardized open-source model packaging format that bridges the gap between model development and deployment. Truss abstracts away framework complexity (PyTorch, TensorFlow, scikit-learn) and enables models to be served on localhost and in production with rapid development and testing loops. It turns ML models into backend APIs, simplifying the process for data scientists by reducing the need for extensive web server code and providing isolated environments for each model to prevent dependency conflicts. Truss also facilitates sharing models as artifacts, making them easier to run on any machine.