Baseten AI Model Deployment and Serving Platform

8/11/2026

This post details the deployment of the GLM-5.2 Fast model, a speed-optimized variant of Z.AI's GLM-5.2 model, on the Baseten platform. It showcases how the model can be used with OpenAI clients through Baseten's inference API, providing example Python code and expected JSON output for chat completions. The GLM-5.2 Fast model is designed for demanding real-time workloads.

8/11/2026

This post introduces 'Dedicated Inference' as a new capability within the Baseten platform, enabling users to run mission-critical inference at massive scale, either in Baseten's cloud or their own infrastructure. It emphasizes the Baseten Inference Stack for this purpose.

8/11/2026

This post showcases Baseten's platform capabilities for LLM inference, focusing on performance and scalability. It highlights how Baseten enables customers to ship LLM-powered apps that scale in any cloud, emphasizing performant, compliant, and reliable inference. The post features several customer logos, implying successful deployments and use cases of Baseten's LLM serving infrastructure.

8/11/2026

This post details the deployment of the NVIDIA Nemotron 3 Ultra model on the Baseten platform. It highlights the model's architecture (550B hybrid Mamba-Transformer MoE with 55B active params, latent MoE routing, multi-token prediction, and 1M token context) and provides example usage for OpenAI-compatible chat completion, including Python code snippets and JSON output structure. It also mentions recommended parameters like temperature=1.0 and top_p=0.95, and the ability to toggle reasoning via chat_template_kwargs.enable_thinking.

8/11/2026

This post introduces and highlights the capability of Baseten to support real-time image generation at infinite scale, emphasizing high performance and cost efficiency. It showcases this capability through customer logos, implying successful deployment and operation of such systems on the Baseten platform.

8/11/2026

This post introduces the Whisper Large V3 model to the Baseten platform, highlighting its implementation achieving 1800x real-time factor for audio transcription. It details recommended hardware configurations (H100 MIG, H100, L4) and concurrency targets for different use cases (balanced, latency-sensitive, cost-sensitive). Example Python code for API usage and expected JSON output for transcription results are provided.

8/4/2026

This post introduces the deployment of NVIDIA Nemotron 3.5 ASR Streaming models on Baseten. It details the architecture of the models (cache-aware FastConformer-RNNT), their performance benchmarks on H100 GPUs (latency, concurrency over WebSocket and gRPC), and accuracy metrics (WER) for both English and multilingual variants. It also highlights the use of NVIDIA NIM for optimized inference and mentions the possibility of fine-tuning via Baseten Training.

7/28/2026

This post serves as a foundational overview of the Baseten platform, detailing its core capabilities for training, deploying, and serving AI models. It introduces Truss for model packaging and deployment, highlights various inference engines optimized for different model architectures, and explains the concept of Chains for orchestrating multi-step AI workflows. The post also covers Baseten's production infrastructure, including autoscaling, multi-cloud capacity management, and observability features. It outlines different user paths for building AI applications, deploying models, and training/fine-tuning.

7/23/2026

Introduces GLM-5.2 Fast, a new Model API tier for GLM-5.2 weights, optimized for per-user throughput for real-time agentic applications. This tier is designed to handle variable, bursty workloads with consistent throughput and low latency. It maintains ease-of-use with OpenAI-compatible API endpoints and a pay-per-token model. The post provides a Python code snippet demonstrating how to switch to the Fast endpoint by changing the model slug.

7/23/2026

This post introduces strategies for choosing and deploying AI models, emphasizing the importance of per-workflow model selection, the value of model switching for reliability and cost, the increasing viability of open-weight models, and the strategic use of fine-tuning for market growth rather than just margin improvement. It highlights the engineering cost and benefits of building flexible model harnesses and the use of A/B testing for subjective output evaluation.

7/23/2026

This post compares NVIDIA H100 and H200 GPUs for AI inference, focusing on hardware specifications, VRAM impact on model size, MIG for smaller models, and async programming for throughput. It provides guidance on choosing between H100 and H200 based on model size, traffic, and budget.

7/16/2026

This post details significant performance optimizations for real-time video generation inference on Baseten, specifically for the Wan 2.2 model. Key contributions include timestep distillation to reduce generation steps, custom kernel engineering for optimized attention mechanisms (e.g., Video Sparse Attention), and NVFP4 quantization for improved memory bandwidth and tensor core throughput. The post also highlights the scalable inference infrastructure required to support this capability, including optimized cold starts via the Baseten Delivery Network and intelligent queuing, as well as the implementation of custom content guardrails for responsible deployment.

7/16/2026

This post introduces the availability of NVIDIA Nemotron 3 Embed models (8B and 1B) on the Baseten platform, enhancing its capabilities for retrieval-augmented generation (RAG) and AI agent applications. It details the technical trade-offs between retrieval accuracy and indexing speed offered by the two model sizes. The post also highlights Baseten's support for fine-tuning these models using the Nemotron Embed fine-tuning recipe and deploying them via Truss for production workloads.

7/15/2026

This post introduces the integration of Thinking Machines Lab's Inkling model, a 975B-parameter multimodal (text, image, audio) open-weight model with a mixture-of-experts architecture, onto the Baseten Platform. It details the technical challenges of serving such a large model, including its significant infrastructure footprint (2TB+ GPU memory for BF16, 600GB for NVFP4). The post highlights how the Baseten Inference Stack, specifically the Baseten Delivery Network for fast cold starts and Multi-cloud Capacity Management for global GPU pooling, enables day-0 support and reliable, scalable serving of Inkling. It also mentions optimizations in vLLM by Inferact for Inkling's deployment.

7/14/2026

This post introduces the integration of Step 3.7 Flash, a 198-billion-parameter sparse MoE vision-language model, into the Baseten Model Library. It details the hardware-efficient deployment strategy using FP8 quantization, enabling the model to be served on 4xH100s instead of the previously recommended 8xH200s/B200s. This optimization significantly reduces cost per token and improves autoscaling. The post also highlights the architectural features of Step 3.7 Flash, including its sparse MoE design, multimodal input capabilities, long context window, and flexible reasoning, and explains how these, combined with hybrid attention and speculative decoding, allow for efficient production traffic serving on the optimized hardware.

7/8/2026

This post details the integration of NVIDIA Nemotron 3 Ultra with LangChain Deep Agents Code (dcode) on the Baseten platform. It outlines the setup process, including installing LangChain dcode with Baseten support and configuring the Baseten API key. A practical example demonstrates building a simple agent for reviewing workflows and flagging compliance errors, showcasing how the model-specific harness profile is automatically applied to achieve high agent accuracy at a lower cost. The post highlights the use of Baseten's model API for Nemotron 3 Ultra and LangChain's dcode for agent development.