Making Kimi K3 tokenization 18x faster for million-token agentic workloads

What this post added

Introduced the Baseten Tokenizer (Basetenkenizer), a Rust-based implementation optimized for long input sequences (up to one million tokens) for Kimi K3. This new tokenizer achieves up to 18x faster tokenization compared to Python's tiktoken for million-token sequences while maintaining exact token ID parity. Key optimizations include specialized pre-tokenization scanners, a stack-resident BPE merge tier, multi-core processing for identical pre-tokens, native typed segments and safe chunking in Rust, zero-copy NumPy ownership transfer, and efficient PyO3 bindings. The implementation also addresses the challenge of distinguishing structural tokens from literal text within prompts.