TechPrufer
Get started
Baseten
Baseten logoBaseten · Feature trail

Kimi K3 Model API Deployment

17 posts · Jul 2025 – Sep 2026

What this trail covers

Baseten continues to optimize its AI model deployment and serving platform, focusing on specific model performance enhancements and architectural evolution. This includes developing specialized APIs for models like GLM-5.2, Kimi K2.5, Kimi K3 and GLM-5.3, prioritizing per-user throughput for real-time agentic applications. Optimizations involve tuning infrastructure for variable, bursty workloads while maintaining tight performance SLAs. The platform supports routing different traffic profiles to specialized inference clusters, and now includes Kimi K2, a 1 trillion parameter MoE model with a novel MuonClip optimizer for stable training and enhanced agentic capabilities, deployed with optimizations for tensor parallelism, KV-cache, and load balancing.

All Baseten trails
  1. DeepSeek-V4.1-Flash: more efficient prefill for coding agentsIntroduces DeepSeek-V4.1-Flash, a 552B-parameter multimodal MoE model with a Causal Encoder-Decoder (CED) architecture. Highlights asymmetric activation parameters (8B for prefill, 16B for decode) for improved cost efficiency in agentic loops. Details KV cache improvements including Compressed Sparse Attention 2 and FP4 KV caching, reducing memory per token. Explains how the Baseten Inference Stack uses NVIDIA Dynamo and KV cache-aware routing for production inference.
  2. Inference engineering for DeepSeek V4 Pro 0813This post details the inference engineering work done to create a production-ready day zero API for DeepSeek V4 Pro 0813. It highlights adjustments made to the inference stack configurations (parallelism, KV cache allocation, prefill-decode worker ratio) based on evolved traffic patterns, longer sequence lengths, and higher cache hit rates. It also covers changes to the frontend and chat template to adhere to new harness expectations and API specifications, and the integration of DSpark speculator for speculative decoding to improve TPS.
  3. 22,580: GPT-2 to Kimi K3, explainedThis post details the architectural evolution from GPT-2 to Kimi K3, highlighting key innovations that enabled massive scale. It explains the fundamental components of GPT-2's transformer architecture, including token and positional embeddings, transformer blocks, and causal self-attention. The post introduces the KV cache as a solution to redundant computation in autoregressive decoding and discusses its memory-bandwidth bottleneck. It then delves into linear attention as an alternative to softmax attention, explaining its reassociable property and the trade-off between expressiveness and efficiency. Finally, it introduces DeltaNet (fast weight programmers) as a mechanism to address information interference in additive caches when sequence length exceeds storage capacity, further optimizing for large-scale models.
  4. How to run Kimi K3 in any harness: routing with Baseten SwitchIntroduces Baseten Switch, a local routing application that allows users to seamlessly integrate and test open-source models (like Kimi K3 and GLM 5.2 Fast) with existing harnesses (Claude Code, Codex, LangChain CLI). It provides a dashboard for monitoring cost and performance metrics of both open and closed-source models, enabling users to make informed decisions about model selection based on workload, cost, and performance. The post details the setup and usage of Baseten Switch within these harnesses, including command-line instructions and recommended patterns for orchestrator and subagent model assignments.
  5. Making Kimi K3 tokenization 18x faster for million-token agentic workloadsIntroduced the Baseten Tokenizer (Basetenkenizer), a Rust-based implementation for Kimi K3 tokenization, achieving up to 18x speedup for million-token sequences compared to Python's tiktoken. The Basetenkenizer consolidates special-token scanning, BPE merging, and safe chunking into a single native call, optimizing for long input sequences in agentic workloads. Key optimizations include specialized pre-tokenization scanners, a stack-resident BPE merge tier, multi-core parallelism for identical pre-tokens, native typed segments with safe chunking in Rust, zero-copy NumPy ownership transfer, and smart-pointer PyO3 bindings. Benchmarks demonstrate significant latency reduction for the serving path, especially with high prefix cache hit rates.
  7. How to build a day-0 API for Kimi K3This post details the technical work required to build a day-0 API for Kimi K3, a 2.8T parameter open frontier model. Key contributions include: provisioning hardware (NVIDIA GB300 NVL72 systems), loading large weights (1.4TB in MXFP4), and bringing up inference engines (vLLM, SGLang) by implementing core modeling code, optimized kernels for KDA and AttnRes, and frontend compatibility for vision input and tool calling. The team validated the inference engine using Kimi Vendor Verifier, focusing on frontend determinism for tool calling and structured outputs. Configuration tuning involved exploring Tensor Parallelism (TP) and Expert Parallelism (EP) across GB300 nodes due to their fast interconnect. Performance optimizations included implementing speculative decoding, disaggregating prefill and decode, and caching KV states. A novel contribution was a custom tokenizer, up to 18x faster than tiktoken for long sequences, which is material to prefill time for models like Kimi K3. The post also touches on deploying at scale to meet anticipated demand.
  8. How we built the new fastest API for GLM-5.2This post details the technical optimizations made to create a significantly faster API for the GLM-5.2 model. Key contributions include: 1. Development of a 'GLM-5.2-Fast' API specifically tuned for reduced latency, trading off throughput. 2. Implementation of latency-focused parallelism strategies (Tensor and Expert Parallelism) for the fast API, distinct from the general API's Attention Data Parallelism. 3. A substantial reduction in the maximum batch size for the fast API to minimize resource contention. 4. General optimizations applied to both APIs, including scheduler improvements, updated NVFP4 weights, and an updated speculative decoding profile. 5. Bug fixes in the inference engine for both quality and performance. The post highlights the trade-off between latency and throughput, with the fast API having 50% higher input and output token prices due to its latency optimizations.
  9. Introducing GLM-5.2 FastIntroduced GLM-5.2 Fast, a new Model API tier optimized for per-user throughput for real-time agentic workloads. This tier preserves the ease-of-use of standard Model APIs with tighter performance SLAs, handling variable and bursty workloads with small variance in throughput and latency. The change is a one-line model slug update, allowing for cost-efficient routing of latency-sensitive agentic loops to Fast and latency-tolerant batch tasks to standard GLM-5.2.
  10. How to optimize LLM inference speed and reduce costs in productionThis post details techniques to optimize LLM inference speed and reduce costs in production, including continuous batching, speculative decoding (draft-target, Medusa, EAGLE-3, N-gram speculation), KV cache optimizations (KV cache-aware routing, CPU offloading), quantization (mixed-precision, rotation-based, curvature-based), kernel optimizations (kernel fusion, attention kernels, asynchronous compute, PDL), intelligent request routing (geo-aware load balancing, LoRA-aware routing), and topology-aware parallelism (Tensor Parallelism, Expert Parallelism).
  11. How we built the world’s fastest API for GLM-5.2This post details the specific optimizations implemented to achieve over 280 TPS for the GLM-5.2 model API. Key contributions include: implementing shared DSA for the GLM-5.2 architecture in the custom inference engine, performing in-house NVFP4 quantization from FP8 weights with equivalent quality on agentic benchmarks, implementing KV-aware routing with NVIDIA Dynamo for improved TTFT and reduced prefill burden, achieving a 2x higher TPS via disaggregated prefill and decode engines using NVIDIA Dynamo, and further improving TPS with GLM-5.2's Multi-Token Prediction heads for speculation.
  12. How we built the fastest Kimi K2.5 on Artificial AnalysisThis post details the optimization of Kimi K2.5 inference by converting INT4 weights to NVFP4 for Blackwell GPU compatibility and implementing a custom EAGLE-3 speculator model trained on Kimi K2.5 hidden states. This achieved 340+ tokens per second, significantly improving inference speed for large reasoning models.
  13. OpenClaw + Kimi K2.5 on Baseten = frontier agent performance with OSSThis post details the integration of the OpenClaw agent framework with Baseten's Kimi K2.5 Model API. It highlights the cost and performance benefits of using Kimi K2.5 on Baseten for agentic tasks compared to closed-source alternatives, showcasing an 8x cost saving and improved performance on agent benchmarks. The post provides a step-by-step guide for users to clone a Baseten-forked OpenClaw repository, install dependencies, and configure it to use the Kimi K2.5 Model API via a Baseten API key, enabling users to deploy and run frontier agent performance locally or in the cloud.
  14. Kimi K2 Thinking at 140+ TPS on NVIDIA BlackwellThis post details the technical work to achieve state-of-the-art performance for Kimi K2 Thinking on NVIDIA Blackwell GPUs. Key contributions include: 1. Converting INT4 quantized weights to NVFP4 by de-quantizing to BF16 and then re-quantizing using NVIDIA TensorRT Model Optimizer, addressing compatibility with Blackwell GPUs. 2. Configuring a mixed Tensor Parallelism (TP) and Expert Parallelism (EP) strategy within the inference engine to effectively utilize an 8xB200 node, balancing latency and throughput. 3. Implementing KV-aware request routing to maximize KV cache re-use, significantly reducing time to first token (TTFT) for use cases like agentic tool use and code generation.
  15. Kimi K2 Explained: The 1 Trillion Parameter ModelThis post introduces Kimi K2, a 1 trillion parameter Mixture-of-Experts (MoE) model from Moonshot Labs, building on the DeepSeek architecture. Key technical innovations include a significantly increased number of specialized experts (384) with fewer attention heads for improved focus, a post-training approach using synthetic agentic data from simulated tool interactions, and the MuonClip optimizer which clips attention logits to prevent training instability and loss spikes. The post also details the production deployment challenges for such a large model, including tensor parallelism, KV-cache optimization, and load balancing, and highlights Baseten's infrastructure for making Kimi K2 accessible with low latency.