LLM Serving Megakernel

1 post · Mar 2025 – Sep 2026

What this trail covers

Cohere has developed and open-sourced a novel serving engine for LLMs built around a decode megakernel. This system significantly improves inference speed and efficiency, particularly at lower batch sizes and longer context lengths, by consolidating the entire forward pass into a single, persistent kernel. It supports advanced features like continuous batching, paged attention, and ragged sequence lengths, offering an OpenAI-compatible endpoint. Model Vault provides secure, scalable, and efficient LLM serving. This post details the challenges and solutions for running LLMs in production, including considerations for hardware, software, and operational aspects to ensure reliable and performant inference.

All Cohere trails

Sep 2026Cohere's North Mini Code Megakernel Serving Engine | CohereThis post details the development and implementation of a fully-fledged LLM serving system built around a decode megakernel, named the North Mini Code Megakernel. It addresses the memory-bound nature of autoregressive decoding at low batch sizes by consolidating the forward pass into a single kernel. Key technical contributions include: reducing launch and synchronization overhead, mitigating wave quantization through parallel transformer layers and aggressive backfilling, dropping false dependencies by enabling fine-grained task synchronization, and implementing weight prefetching to overlap weight loading with previous task execution. The system supports continuous batching, paged attention, and ragged sequence lengths, and is demonstrated to outperform vLLM by 1.58x at batch size 1 and maintain performance advantages across batch sizes and long contexts.
Aug 2026Cohere Releases Command A+ for Sovereign AI | CohereThis post introduces Command A+, a new open-source Mixture-of-Experts (MoE) model designed for sovereign AI deployments. It highlights the model's efficiency with a minimal compute footprint (218B parameters, 24B active per prompt) enabling high-performance inference on limited hardware (e.g., two H100s or one B200 GPU). The model is engineered for enterprise workloads like RAG, SQL generation, and financial document analysis, with multimodal capabilities for visual data. It also emphasizes strong multilingual performance across 48 languages and sovereign design principles including full transparency, on-premises deployment, regulatory alignment, and no vendor lock-in.
Jul 2026Production-Ready W4A8: vLLM Integration Explained | CohereThis post details the development and integration of production-ready W4A8 GEMM kernels for NVIDIA Hopper GPUs within the vLLM framework. It addresses the challenges of achieving high performance with mixed precision (INT4 weights, FP8 activations) by implementing a lookup-table (LUT) based dequantization strategy and a custom CUDA kernel for faster pre-processing. The post also describes techniques to maintain model quality with W4A8 quantization, including quantizing group scales to FP8 with outer vector scaling and compensating for clipping by scaling FP8 scales and per-channel scales. Furthermore, it introduces token masking support to AWQ for more accurate calibration data in agentic workflows, particularly for long-context scenarios.
Jul 2026Why MoE Models Get More From Speculative Decoding | CohereThis post analyzes the interaction of Mixture-of-Experts (MoE) models with speculative decoding (SD). It validates the prediction of a non-monotonic speedup curve for MoE models, showing gains that first increase and then decline with batch size, contrasting with the monotonic decrease seen in dense models. The analysis attributes this to arithmetic intensity and identifies three batch-size regimes (low, moderate, high) influencing SD gains. It proposes co-design principles for model sparsity and speculative decoding based on target batch size. Furthermore, it quantifies the impact of temporal correlation in expert routing on SD verification cost, demonstrating that adjacent tokens tend to activate overlapping experts, which reduces the number of unique experts loaded and improves efficiency. The post provides empirical data on expert selection probabilities and expected active experts, showing that real routing concentrates experts more than theoretical baselines.
Nov 2025Making data transfer in LLM systems faster | CohereThis post details optimizations for data transfer within LLM systems, focusing on reducing latency and improving throughput for faster data movement between components. It introduces techniques to accelerate data loading and processing, which are critical for efficient LLM inference and overall system performance.
Advertisement
Mar 2025Running Large Language Models in ProductionThis post details the challenges and solutions for running Large Language Models (LLMs) in production. It covers considerations for hardware selection (e.g., GPU types, memory), software stack optimization (e.g., inference engines, quantization techniques), and operational aspects like monitoring, scaling, and cost management. Specific techniques discussed include efficient batching strategies, memory management for large models, and strategies for handling variable request loads to ensure reliable and performant inference.