Kimi K3 Model API Deployment

8/11/2026

This post introduces the Baseten Inference Runtime, highlighting its capability to achieve frontier performance with lowest latency and highest throughput for AI model deployment. It implies ongoing optimization efforts for model serving, building upon previous work with models like Kimi K3.

8/11/2026

Introduces DeepSeek-V4-Flash-0731, an open-weight 284B MoE (13B active) model with a 1M context window and selectable reasoning effort, tuned for coding, chat, and agent workflows. Highlights its efficiency-oriented design, hybrid attention architecture for long-context processing, and configurable reasoning-effort levels for trading latency and token usage for deliberation. The 0731 release specifically focuses on agentic performance gains on coding, tool-use, and automation benchmarks.

8/11/2026

Introduces Kimi K2.7 Code, an open-source coding LLM with improved performance on coding benchmarks (Kimi Code Bench v2, Program Bench, MLS Bench Lite) and reduced reasoning token usage compared to its predecessor. Provides example usage with OpenAI clients and demonstrates JSON output including usage statistics.

7/30/2026

This post explains the architectural evolution from GPT-2 to Kimi K3, focusing on advancements in attention mechanisms. It details the inefficiency of standard decoder-only generation without caching, introduces the KV cache and its memory-bandwidth bottleneck, and then explains linear attention as a solution that replaces the growing KV cache with a fixed DxD state, reducing memory traffic for long sequences. It also introduces DeltaNet (Fast Weight Programmers) as a method to address information interference in linear attention when sequence length exceeds capacity, by learning to dynamically interact with and selectively update memory contents.

7/30/2026

Introduces Baseten Switch, a local routing application that allows users to seamlessly integrate and test open-source models (like Kimi K3 and GLM 5.2 Fast) with existing harnesses (Claude Code, Codex, LangChain CLI). It provides a dashboard for monitoring cost and performance metrics of both open and closed-source models, enabling users to make informed decisions about model selection based on workload, cost, and performance. The post details the setup and usage of Baseten Switch within these harnesses, including command-line instructions and recommended patterns for orchestrator and subagent model assignments.

7/29/2026

Introduces 'Baseten for Model Labs' as a new distribution platform for closed-weight models. This platform provides managed inference infrastructure, handles billing, API keys, compliance, and compute procurement. It aims to increase visibility for model labs by listing their models in the Baseten Model Library, offering strong IP protection, and providing go-to-market support. The post also highlights partnerships with 15 model labs and their specific model offerings, such as Cartesia's Sonic (TTS) and Ink (STT), Gradiuim's real-time speech models, Inception's Mercury 2 diffusion LLM, NVIDIA's Nemotron ASR and BioNeMo, PyannoteAI's speaker intelligence model, SID.ai's specialized search LLM, Subconscious's long-horizon AI agent systems, and Synthefy's Nori tabular foundation model.

7/27/2026

Introduced the Baseten Tokenizer (Basetenkenizer), a Rust-based implementation optimized for long input sequences (up to one million tokens) for Kimi K3. This new tokenizer achieves up to 18x faster tokenization compared to Python's tiktoken for million-token sequences while maintaining exact token ID parity. Key optimizations include specialized pre-tokenization scanners, a stack-resident BPE merge tier, multi-core processing for identical pre-tokens, native typed segments and safe chunking in Rust, zero-copy NumPy ownership transfer, and efficient PyO3 bindings. The implementation also addresses the challenge of distinguishing structural tokens from literal text within prompts.

7/27/2026

This post describes the technical work involved in enabling day-0 support for the Kimi K3 model on Baseten's Model APIs. It details the challenges and solutions for running a 2.8T parameter model with novel architectural features like KDA, AttnRes, and Stable LatentMoE. Specific contributions include integrating with vLLM and SGLang, validating model fidelity with Kimi Vendor Verifier, optimizing inference configurations, developing a custom tokenizer that is up to 18x faster than tiktoken for long sequences, and implementing performance enhancements like speculative decoding and disaggregation of prefill and decode stages.

7/26/2026

This post details the development of a new, significantly faster API for the GLM-5.2 model, achieving 601 tokens per second. It introduces a 'GLM-5.2-Fast' API that prioritizes latency over throughput by using Tensor and Expert Parallelism instead of Attention Data Parallelism, and reducing the max batch size. The post also mentions general optimizations to the scheduler, NVFP4 weights, speculative decoding profiles, and bug fixes in the inference engine.

7/23/2026

This post details techniques to optimize LLM inference speed and reduce costs in production, including continuous batching, speculative decoding (draft-target, Medusa, EAGLE-3, N-gram speculation), KV cache optimizations (KV cache-aware routing, CPU offloading), quantization (mixed-precision, rotation-based, curvature-based), kernel optimizations (kernel fusion, attention kernels, asynchronous compute, PDL), intelligent request routing (geo-aware load balancing, LoRA-aware routing), and topology-aware parallelism (Tensor Parallelism, Expert Parallelism).