Cohere's North Mini Code Megakernel Serving Engine | Cohere

This post details the development and implementation of a fully-fledged LLM serving system built around a decode megakernel, named the North Mini Code Megakernel. It addresses the memory-bound nature of autoregressive decoding at low batch sizes by consolidating the forward pass into a single kernel. Key technical contributions include: reducing launch and synchronization overhead, mitigating wave quantization through parallel transformer layers and aggressive backfilling, dropping false dependencies by enabling fine-grained task synchronization, and implementing weight prefetching to overlap weight loading with previous task execution. The system supports continuous batching, paged attention, and ragged sequence lengths, and is demonstrated to outperform vLLM by 1.58x at batch size 1 and maintain performance advantages across batch sizes and long contexts.