TechPrufer
Sign inGet started

Blogs›Crusoe›Fast Tokenization

Fast Tokenization

Fast Tokenization

2
posts
2026

fastokens v2 introduces native tiktoken support, a regex-free Unicode scanner for pretokenization, and a rebuilt BPE core for improved serving latency and bulk throughput. It now supports tiktoken.model files directly, enabling compatibility with models like Kimi K3. Performance gains are significant, with up to 10-46x speedups over HuggingFace tokenizers and 10x over tiktoken on specific workloads. Features for LLM serving include a prefix cache for shared prompts and an `encode_batch_flat` function. This post discusses the broader implications of agentic inference on tokenomics, emphasizing cost per token as the ultimate measure of inference efficiency. It highlights how agentic workloads, with their multi-step plans and stateful context, can consume significantly more tokens per task, necessitating a deeper evaluation of infrastructure beyond GPU/hour and FLOPS/dollar. The post details the 'inference iceberg' metaphor, distinguishing between visible metrics (GPU specs) and underlying factors (power, cooling, networking, software stack) that impact real-world token output and cost. It introduces new metrics like tokens per watt and cost per completed task, and discusses the role of physical infrastructure and the open-source software stack in optimizing agentic AI inference. Crusoe's vertically-integrated AI factories and early adoption of NVIDIA's DSX Platform are presented as key enablers for achieving low cost per token in agentic workloads. The post also mentions Crusoe's Managed Inference and Serverless Fine-Tuning services for deploying and customizing open agentic AI systems.

2026

fastokens v2: native tiktoken support, faster BPE core

8/11/2026

This post details the v2 release of fastokens, which adds native support for tiktoken models, replaces the pretokenization regex with a hand-written Unicode scanner, and rebuilds the BPE core for performance. Key improvements include direct loading of tiktoken.model files, a faster pretokenizer with byte checks and tables for ASCII and non-ASCII characters, and an optimized BPE encode loop leveraging cache hits. For serving, it introduces a prefix cache for shared prompts and `encode_batch_flat` for efficient batch output. All changes are validated bit-for-bit against reference tokenizers.

Tokenomics in the Age of Agentic Inference

8/11/2026

This post expands on the concept of tokenization by discussing its economic implications in the context of agentic AI inference. It introduces the 'inference iceberg' metaphor to illustrate that cost per token is the true measure of inference efficiency, driven by factors beyond raw compute. It details how agentic workloads, with their multi-step execution and stateful context, dramatically increase token consumption, necessitating a holistic evaluation of physical infrastructure (power, cooling, networking) and the open-source software stack (serving frameworks, models). The post highlights Crusoe's approach to building vertically-integrated AI factories and its adoption of NVIDIA's DSX Platform as strategies to optimize for low cost per token in agentic AI. It also mentions Crusoe's Managed Inference and Serverless Fine-Tuning services as solutions for deploying and customizing agentic AI systems.