How to build a day-0 API for Kimi K3

This post details the technical work required to build a day-0 API for Kimi K3, a 2.8T parameter open frontier model. Key contributions include: provisioning hardware (NVIDIA GB300 NVL72 systems), loading large weights (1.4TB in MXFP4), and bringing up inference engines (vLLM, SGLang) by implementing core modeling code, optimized kernels for KDA and AttnRes, and frontend compatibility for vision input and tool calling. The team validated the inference engine using Kimi Vendor Verifier, focusing on frontend determinism for tool calling and structured outputs. Configuration tuning involved exploring Tensor Parallelism (TP) and Expert Parallelism (EP) across GB300 nodes due to their fast interconnect. Performance optimizations included implementing speculative decoding, disaggregating prefill and decode, and caching KV states. A novel contribution was a custom tokenizer, up to 18x faster than tiktoken for long sequences, which is material to prefill time for models like Kimi K3. The post also touches on deploying at scale to meet anticipated demand.